You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "cetra3 (via GitHub)" <gi...@apache.org> on 2026/07/14 11:18:44 UTC
[PR] GC & Deduplicate String View on Spill [datafusion]
cetra3 opened a new pull request, #23565:
URL: https://github.com/apache/datafusion/pull/23565
## Which issue does this PR close?
- Closes https://github.com/apache/datafusion/issues/23564
## Rationale for this change
Spill files inflate string views via GC. I.e, a highly deduped string view can inflate considerably causing more memory and disk pressure.
## What changes are included in this PR?
There is a new method added in the spill path to GC & Dedup string views
## Are these changes tested?
Yes they are covered by existing tests.
**However:** I would like to make sure we run benchmarks with external sorting to see if this is a considerable regression. My instinct will tell me that, counterintuitively, this might be a performance *increase* when externally sorting since there is less bytes being moved around.
## Are there any user-facing changes?
No this is an internal method change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783311204
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783079245)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783079245-2590-h8mmt 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784256829
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783903325)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 105.1Β±0.37ms ? ?/sec 1.02 106.7Β±0.92ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 101.0Β±0.92ms ? ?/sec 1.00 101.5Β±0.83ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 99.3Β±0.43ms ? ?/sec 1.01 100.6Β±1.14ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 123.4Β±1.39ms ? ?/sec 1.03 126.9Β±1.02ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 133.8Β±4.87ms ? ?/sec 1.00 134.2Β±3.29ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.2s |
| Peak memory | 772.5 MiB |
| Avg memory | 60.1 MiB |
| CPU user | 64.6s |
| CPU sys | 30.5s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 370.2s |
| Peak memory | 755.1 MiB |
| Avg memory | 59.2 MiB |
| CPU user | 65.2s |
| CPU sys | 30.8s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783700534
Benchmark for [this request](https://github.com/apache/datafusion/pull/23565#issuecomment-5783079245) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `spill_views`
<details><summary>Runner log (last 40 lines)</summary>
```
Compiling datafusion-datasource-json v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-json)
Compiling datafusion-datasource-arrow v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-arrow)
Compiling datafusion-catalog-listing v55.1.0 (/workspace/datafusion-branch/datafusion/catalog-listing)
Compiling datafusion-functions-table v55.1.0 (/workspace/datafusion-branch/datafusion/functions-table)
Compiling datafusion-datasource-parquet v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-parquet)
Compiling datafusion-physical-optimizer v55.1.0 (/workspace/datafusion-branch/datafusion/physical-optimizer)
Compiling datafusion v55.1.0 (/workspace/datafusion-branch/datafusion/core)
Compiling datafusion-proto v55.1.0 (/workspace/datafusion-branch/datafusion/proto)
Compiling datafusion-benchmarks v55.1.0 (/workspace/datafusion-branch/benchmarks)
Finished `bench` profile [optimized] target(s) in 10m 29s
Running benches/sql.rs (/workspace/datafusion-branch/target/release/deps/sql-2af2193982da74c5)
Gnuplot not found, using plotters backend
Benchmarking spill_views/q01_sort_string_1_distinct_repeated
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Collecting 10 samples in estimated 5.4214 s (60 iterations)
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Analyzing
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Collecting 10 samples in estimated 5.9142 s (60 iterations)
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Collecting 10 samples in estimated 5.9238 s (60 iterations)
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q04_sort_string_all_distinct_distinct
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Collecting 10 samples in estimated 5.4204 s (30 iterations)
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Analyzing
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Collecting 10 samples in estimated 5.9546 s (40 iterations)
thread 'main' (16297) panicked at benchmarks/src/sql_benchmark_runner.rs:224:25:
Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted("Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\n FinalHashAggregateStream[1]#5932(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[2]#5934(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[0]#5941(can spill: true) consumed 27.1 MB, peak 27.1 MB,\n RepartitionExec[0]#5939(can spill: true) consumed 4.8 MB, peak 4.8 MB,\n RepartitionExec[2]#5942(can spill: true) consumed 2.4 MB, peak 2.4 MB.\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' (16297) panicked at benchmarks/benches/sql.rs:55:27:
failed to run SQL benchmarks: Execution("criterion benchmark failed: Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted(\"Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\\n FinalHashAggregateStream[1]#5932(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[2]#5934(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[0]#5941(can spill: true) consumed 27.1 MB, peak 27.1 MB,\\n RepartitionExec[0]#5939(can spill: true) consumed 4.8 MB, peak 4.8 MB,\\n RepartitionExec[2]#5942(can spill: true) consumed 2.4 MB, peak 2.4 MB.\\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)\")")
error: bench failed, to rerun pass `--bench sql`
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863158797
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111096)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863111096-2861-clwzz 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863478656
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863112345)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.26 107.8Β±1.07ms ? ?/sec 1.00 85.4Β±0.49ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.08 103.3Β±0.60ms ? ?/sec 1.00 95.5Β±0.62ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.07 101.8Β±0.74ms ? ?/sec 1.00 95.2Β±0.66ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 128.8Β±0.75ms ? ?/sec 1.02 131.7Β±0.98ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 135.1Β±4.58ms ? ?/sec 1.01 136.8Β±3.71ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69 (merge-base)`) | 102.6 MiB | 790.8 MiB | 688.2 MiB | 7.7Γ |
| spill_views | changed (`spill-dedup-view-arrays`) | 102.6 MiB | 750.0 MiB | 647.4 MiB | 7.3Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 385.2s |
| Peak memory | 790.8 MiB |
| Avg memory | 55.7 MiB |
| CPU user | 63.5s |
| CPU sys | 30.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 640.2s |
| Peak memory | 750.0 MiB |
| Avg memory | 41.7 MiB |
| CPU user | 80.9s |
| CPU sys | 22.9s |
| Peak spill | 73.8 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789127530
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
changed:
ref: 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789377593
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789121802)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.19 105.5Β±1.08ms ? ?/sec 1.00 88.4Β±0.48ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.02 100.1Β±0.74ms ? ?/sec 1.00 97.8Β±0.52ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 98.8Β±0.90ms ? ?/sec 1.00 98.4Β±0.51ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 126.6Β±1.00ms ? ?/sec 1.43 180.7Β±2.68ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 111.2Β±0.85ms ? ?/sec 1.05 117.1Β±0.82ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 565.2s |
| Peak memory | 821.5 MiB |
| Avg memory | 40.8 MiB |
| CPU user | 66.9s |
| CPU sys | 29.0s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 565.3s |
| Peak memory | 817.2 MiB |
| Avg memory | 44.7 MiB |
| CPU user | 78.6s |
| CPU sys | 18.7s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5790071725
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789851638)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 7570366fd929daf9ced744bb8397686b50565b18 (7570366fd929daf9ced744bb8397686b50565b18) to 7570366fd929daf9ced744bb8397686b50565b18 [diff](https://github.com/apache/datafusion/compare/7570366fd929daf9ced744bb8397686b50565b18..7570366fd929daf9ced744bb8397686b50565b18)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
changed:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 106.8Β±0.81ms ? ?/sec 1.02 108.4Β±1.05ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 102.3Β±0.51ms ? ?/sec 1.02 104.0Β±1.02ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 101.0Β±0.57ms ? ?/sec 1.02 102.8Β±0.76ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 127.1Β±0.74ms ? ?/sec 1.02 130.0Β±1.16ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 132.8Β±7.06ms ? ?/sec 1.04 138.3Β±3.87ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `7570366fd929daf9ced744bb8397686b50565b18` | Changed: `7570366fd929daf9ced744bb8397686b50565b18`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`7570366fd929daf9ced744bb8397686b50565b18`) | 102.6 MiB | 726.9 MiB | 624.3 MiB | 7.1Γ |
| spill_views | changed (`7570366fd929daf9ced744bb8397686b50565b18`) | 102.6 MiB | 763.6 MiB | 661.0 MiB | 7.4Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 425.3s |
| Peak memory | 726.9 MiB |
| Avg memory | 49.7 MiB |
| CPU user | 63.6s |
| CPU sys | 29.9s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 420.2s |
| Peak memory | 763.6 MiB |
| Avg memory | 52.0 MiB |
| CPU user | 65.7s |
| CPU sys | 30.9s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789376636
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789123708)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.16 105.3Β±2.86ms ? ?/sec 1.00 90.4Β±1.86ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.01 99.9Β±0.78ms ? ?/sec 1.00 99.4Β±2.08ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.03 100.6Β±4.07ms ? ?/sec 1.00 98.0Β±1.00ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 126.3Β±1.76ms ? ?/sec 1.38 174.3Β±2.87ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.01 120.4Β±3.65ms ? ?/sec 1.00 118.9Β±2.66ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 565.2s |
| Peak memory | 842.1 MiB |
| Avg memory | 41.4 MiB |
| CPU user | 68.8s |
| CPU sys | 30.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 580.3s |
| Peak memory | 773.2 MiB |
| Avg memory | 42.4 MiB |
| CPU user | 79.2s |
| CPU sys | 19.4s |
| Peak spill | 80.0 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248827939
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "256M"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248943145
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5248827939)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "256M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 121.46 ms β 121.23 ms β no change β
β Q2 β 117.93 ms β 116.89 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 179.42 ms β 176.85 ms β no change β
β Q5 β 319.50 ms β 315.05 ms β no change β
β Q6 β 332.11 ms β 335.62 ms β no change β
β Q7 β FAIL β FAIL β incomparable β
β Q8 β FAIL β FAIL β incomparable β
β Q9 β FAIL β FAIL β incomparable β
β Q10 β FAIL β FAIL β incomparable β
β Q11 β FAIL β FAIL β incomparable β
βββββββββ΄ββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1070.42ms β
β Total Time (spill-dedup-view-arrays) β 1065.63ms β
β Average Time (HEAD) β 214.08ms β
β Average Time (spill-dedup-view-arrays) β 213.13ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 5 β
β Queries with Failure β 6 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 121.46 / 122.26 Β±0.89 / 123.92 ms β 121.23 / 122.02 Β±0.74 / 123.11 ms β no change β
β Q2 β 117.93 / 120.64 Β±2.59 / 125.51 ms β 116.89 / 118.47 Β±1.28 / 119.67 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 179.42 / 183.07 Β±3.03 / 187.73 ms β 176.85 / 180.64 Β±2.20 / 182.97 ms β no change β
β Q5 β 319.50 / 332.54 Β±6.73 / 338.12 ms β 315.05 / 327.28 Β±8.44 / 339.33 ms β no change β
β Q6 β 332.11 / 334.52 Β±2.43 / 338.76 ms β 335.62 / 341.01 Β±5.78 / 349.72 ms β no change β
β Q7 β FAIL β FAIL β incomparable β
β Q8 β FAIL β FAIL β incomparable β
β Q9 β FAIL β FAIL β incomparable β
β Q10 β FAIL β FAIL β incomparable β
β Q11 β FAIL β FAIL β incomparable β
βββββββββ΄ββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 1093.04ms β
β Total Time (spill-dedup-view-arrays) β 1089.41ms β
β Average Time (HEAD) β 218.61ms β
β Average Time (spill-dedup-view-arrays) β 217.88ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 5 β
β Queries with Failure β 6 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `a9b61ab (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 240.8 MiB | 241.2 MiB | +0.1% |
| 2 | 248.5 MiB | 250.5 MiB | +0.8% |
| 3 | 254.4 MiB | 254.9 MiB | +0.2% |
| 4 | 250.9 MiB | 249.7 MiB | -0.5% |
| 5 | 247.5 MiB | 250.6 MiB | +1.3% |
| 6 | 249.5 MiB | 245.1 MiB | -1.8% |
| 7 | 259.5 MiB | 253.3 MiB | -2.4% |
| 8 | 255.7 MiB | 255.5 MiB | -0.1% |
| 9 | 256.9 MiB | 257.9 MiB | +0.4% |
| 10 | 255.1 MiB | 255.8 MiB | +0.3% |
| 11 | 256.2 MiB | 254.1 MiB | -0.8% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`a9b61ab (merge-base)`) | 259.5 MiB | 1.2 GiB | 1011.7 MiB | 4.9Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 257.9 MiB | 1.3 GiB | 1.0 GiB | 5.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 10.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 577.1 MiB |
| CPU user | 28.4s |
| CPU sys | 5.5s |
| Peak spill | 0 B |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 10.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 569.4 MiB |
| CPU user | 23.1s |
| CPU sys | 3.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969201661
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 39.76 / 42.28 Β±2.36 / 46.10 ms β 38.31 / 39.36 Β±1.13 / 41.55 ms β +1.07x faster β
β QQuery 2 β 20.40 / 20.67 Β±0.15 / 20.84 ms β 19.38 / 19.75 Β±0.21 / 19.97 ms β no change β
β QQuery 3 β 32.53 / 33.62 Β±1.17 / 35.85 ms β 31.08 / 32.92 Β±1.19 / 33.92 ms β no change β
β QQuery 4 β 18.39 / 18.63 Β±0.23 / 19.06 ms β 17.77 / 18.19 Β±0.69 / 19.57 ms β no change β
β QQuery 5 β 41.46 / 43.24 Β±1.42 / 45.74 ms β 38.81 / 40.97 Β±1.09 / 41.75 ms β +1.06x faster β
β QQuery 6 β 16.97 / 18.09 Β±1.77 / 21.61 ms β 16.29 / 16.61 Β±0.29 / 17.14 ms β +1.09x faster β
β QQuery 7 β 50.62 / 51.41 Β±0.78 / 52.72 ms β 44.85 / 46.22 Β±0.85 / 47.26 ms β +1.11x faster β
β QQuery 8 β 46.31 / 46.75 Β±0.27 / 47.05 ms β 43.39 / 43.60 Β±0.14 / 43.81 ms β +1.07x faster β
β QQuery 9 β 53.80 / 56.20 Β±1.64 / 58.08 ms β 49.75 / 50.35 Β±0.70 / 51.73 ms β +1.12x faster β
β QQuery 10 β 46.12 / 47.10 Β±1.29 / 49.53 ms β 42.49 / 42.62 Β±0.07 / 42.71 ms β +1.11x faster β
β QQuery 11 β 14.76 / 15.23 Β±0.65 / 16.51 ms β 13.75 / 14.09 Β±0.33 / 14.72 ms β +1.08x faster β
β QQuery 12 β 25.58 / 27.38 Β±2.28 / 31.51 ms β 23.98 / 24.51 Β±0.28 / 24.82 ms β +1.12x faster β
β QQuery 13 β 34.22 / 35.84 Β±1.47 / 38.41 ms β 32.56 / 34.22 Β±1.70 / 37.26 ms β no change β
β QQuery 14 β 25.80 / 26.05 Β±0.38 / 26.81 ms β 23.89 / 24.05 Β±0.12 / 24.22 ms β +1.08x faster β
β QQuery 15 β 33.68 / 33.92 Β±0.20 / 34.18 ms β 31.49 / 31.64 Β±0.22 / 32.08 ms β +1.07x faster β
β QQuery 16 β 14.91 / 15.07 Β±0.11 / 15.20 ms β 13.92 / 14.27 Β±0.20 / 14.54 ms β +1.06x faster β
β QQuery 17 β 78.08 / 81.94 Β±3.68 / 87.54 ms β 73.34 / 74.14 Β±0.46 / 74.62 ms β +1.11x faster β
β QQuery 18 β 62.08 / 64.01 Β±1.86 / 67.06 ms β 60.27 / 61.21 Β±0.60 / 62.16 ms β no change β
β QQuery 19 β 34.21 / 34.48 Β±0.19 / 34.70 ms β 33.15 / 33.43 Β±0.44 / 34.30 ms β no change β
β QQuery 20 β 33.32 / 34.07 Β±1.08 / 36.21 ms β 32.50 / 32.78 Β±0.24 / 33.16 ms β no change β
β QQuery 21 β 56.18 / 58.90 Β±1.57 / 61.01 ms β 54.99 / 57.40 Β±1.66 / 59.99 ms β no change β
β QQuery 22 β 14.28 / 14.47 Β±0.17 / 14.77 ms β 14.15 / 14.29 Β±0.08 / 14.39 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 819.37ms β
β Total Time (spill-dedup-view-arrays) β 766.60ms β
β Average Time (HEAD) β 37.24ms β
β Average Time (spill-dedup-view-arrays) β 34.85ms β
β Queries Faster β 13 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 672.2 MiB |
| CPU user | 23.5s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 531.0 MiB |
| CPU user | 22.2s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969073567
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4969050947-1048-jrc8t 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (25cc81b71537b63a872d994c8b0512b0bb715fa0) to f755cb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/f755cb44fa33c69837a8774be862fbfe01e7a100..25cc81b71537b63a872d994c8b0512b0bb715fa0) using: tpcds
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248855798
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5248827939)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5248827939-1538-nxfvd 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "256M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5249003089
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789152679
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789123708)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789123708-2625-j2t85 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783139231
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783076971-2587-xqz2n 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783731694
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982
run benchmarks
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783697679
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783081490)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 104.3Β±0.59ms ? ?/sec 1.01 104.9Β±0.59ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 100.4Β±0.49ms ? ?/sec 1.00 100.3Β±0.50ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 98.1Β±0.63ms ? ?/sec 1.01 99.0Β±0.72ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.01 123.3Β±1.34ms ? ?/sec 1.00 122.4Β±0.62ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.01 135.7Β±3.62ms ? ?/sec 1.00 134.4Β±3.47ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 530.2s |
| Peak memory | 722.7 MiB |
| Avg memory | 40.6 MiB |
| CPU user | 65.3s |
| CPU sys | 30.5s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 665.2s |
| Peak memory | 734.1 MiB |
| Avg memory | 32.0 MiB |
| CPU user | 64.8s |
| CPU sys | 30.4s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784130230
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783730184)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.18 106.8Β±1.79ms ? ?/sec 1.00 90.8Β±0.80ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.02 102.4Β±1.76ms ? ?/sec 1.00 100.3Β±0.83ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.02 102.0Β±0.94ms ? ?/sec 1.00 100.5Β±0.56ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 360.2s |
| Peak memory | 412.6 MiB |
| Avg memory | 27.5 MiB |
| CPU user | 29.3s |
| CPU sys | 18.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 365.2s |
| Peak memory | 493.9 MiB |
| Avg memory | 32.2 MiB |
| CPU user | 37.6s |
| CPU sys | 8.5s |
| Peak spill | 30.6 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784129955
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783731694)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 104.7Β±0.77ms ? ?/sec 1.00 104.8Β±0.71ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 100.4Β±0.99ms ? ?/sec 1.00 100.2Β±0.51ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 98.6Β±0.79ms ? ?/sec 1.00 98.9Β±0.74ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 355.2s |
| Peak memory | 432.4 MiB |
| Avg memory | 29.3 MiB |
| CPU user | 30.2s |
| CPU sys | 18.0s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 360.2s |
| Peak memory | 445.5 MiB |
| Avg memory | 28.1 MiB |
| CPU user | 29.1s |
| CPU sys | 17.4s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783315671
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 6.10 ms β 6.07 ms β no change β
β QQuery 2 β 83.35 ms β 82.53 ms β no change β
β QQuery 3 β 30.07 ms β 29.69 ms β no change β
β QQuery 4 β 507.95 ms β 491.25 ms β no change β
β QQuery 5 β 52.03 ms β 51.85 ms β no change β
β QQuery 6 β 36.99 ms β 36.67 ms β no change β
β QQuery 7 β 75.64 ms β 75.10 ms β no change β
β QQuery 8 β 37.38 ms β 36.92 ms β no change β
β QQuery 9 β 53.79 ms β 52.96 ms β no change β
β QQuery 10 β 62.85 ms β 62.56 ms β no change β
β QQuery 11 β 304.08 ms β 305.38 ms β no change β
β QQuery 12 β 31.19 ms β 29.96 ms β no change β
β QQuery 13 β 124.04 ms β 121.51 ms β no change β
β QQuery 14 β 408.01 ms β 401.59 ms β no change β
β QQuery 15 β 58.82 ms β 58.35 ms β no change β
β QQuery 16 β 6.91 ms β 6.94 ms β no change β
β QQuery 17 β 79.90 ms β 79.89 ms β no change β
β QQuery 18 β 107.36 ms β 106.76 ms β no change β
β QQuery 19 β 42.16 ms β 41.97 ms β no change β
β QQuery 20 β 36.82 ms β 36.79 ms β no change β
β QQuery 21 β 17.44 ms β 17.46 ms β no change β
β QQuery 22 β 66.70 ms β 67.28 ms β no change β
β QQuery 23 β 330.10 ms β 329.26 ms β no change β
β QQuery 24 β 212.45 ms β 204.15 ms β no change β
β QQuery 25 β 112.13 ms β 109.64 ms β no change β
β QQuery 26 β 50.14 ms β 48.76 ms β no change β
β QQuery 27 β 6.85 ms β 6.44 ms β +1.06x faster β
β QQuery 28 β 61.95 ms β 57.75 ms β +1.07x faster β
β QQuery 29 β 98.62 ms β 96.28 ms β no change β
β QQuery 30 β 35.10 ms β 35.16 ms β no change β
β QQuery 31 β 114.96 ms β 114.77 ms β no change β
β QQuery 32 β 21.86 ms β 21.52 ms β no change β
β QQuery 33 β 39.40 ms β 38.58 ms β no change β
β QQuery 34 β 10.70 ms β 10.44 ms β no change β
β QQuery 35 β 78.32 ms β 78.25 ms β no change β
β QQuery 36 β 6.30 ms β 6.26 ms β no change β
β QQuery 37 β 7.43 ms β 7.58 ms β no change β
β QQuery 38 β 65.75 ms β 67.41 ms β no change β
β QQuery 39 β 79.39 ms β 78.22 ms β no change β
β QQuery 40 β 26.31 ms β 25.85 ms β no change β
β QQuery 41 β 12.41 ms β 12.10 ms β no change β
β QQuery 42 β 25.12 ms β 25.24 ms β no change β
β QQuery 43 β 5.96 ms β 5.57 ms β +1.07x faster β
β QQuery 44 β 10.43 ms β 9.79 ms β +1.07x faster β
β QQuery 45 β 47.24 ms β 45.16 ms β no change β
β QQuery 46 β 12.78 ms β 12.36 ms β no change β
β QQuery 47 β 264.79 ms β 246.71 ms β +1.07x faster β
β QQuery 48 β 99.19 ms β 98.18 ms β no change β
β QQuery 49 β 73.44 ms β 72.58 ms β no change β
β QQuery 50 β 60.34 ms β 61.54 ms β no change β
β QQuery 51 β 95.74 ms β 98.06 ms β no change β
β QQuery 52 β 25.15 ms β 25.78 ms β no change β
β QQuery 53 β 29.91 ms β 30.42 ms β no change β
β QQuery 54 β 55.83 ms β 56.39 ms β no change β
β QQuery 55 β 24.85 ms β 24.86 ms β no change β
β QQuery 56 β 40.39 ms β 40.59 ms β no change β
β QQuery 57 β 181.07 ms β 179.05 ms β no change β
β QQuery 58 β 114.34 ms β 113.42 ms β no change β
β QQuery 59 β 119.87 ms β 119.29 ms β no change β
β QQuery 60 β 40.79 ms β 40.42 ms β no change β
β QQuery 61 β 12.25 ms β 12.31 ms β no change β
β QQuery 62 β 45.67 ms β 45.39 ms β no change β
β QQuery 63 β 30.83 ms β 30.17 ms β no change β
β QQuery 64 β 374.19 ms β 368.65 ms β no change β
β QQuery 65 β 134.35 ms β 134.52 ms β no change β
β QQuery 66 β 80.02 ms β 79.74 ms β no change β
β QQuery 67 β 263.33 ms β 267.33 ms β no change β
β QQuery 68 β 12.72 ms β 12.17 ms β no change β
β QQuery 69 β 58.56 ms β 56.96 ms β no change β
β QQuery 70 β 107.62 ms β 104.98 ms β no change β
β QQuery 71 β 36.22 ms β 35.99 ms β no change β
β QQuery 72 β 1859.33 ms β 1817.92 ms β no change β
β QQuery 73 β 10.47 ms β 10.88 ms β no change β
β QQuery 74 β 176.46 ms β 190.90 ms β 1.08x slower β
β QQuery 75 β 144.20 ms β 147.67 ms β no change β
β QQuery 76 β 35.17 ms β 36.34 ms β no change β
β QQuery 77 β 62.91 ms β 64.03 ms β no change β
β QQuery 78 β 177.07 ms β 173.90 ms β no change β
β QQuery 79 β 70.15 ms β 69.34 ms β no change β
β QQuery 80 β 99.72 ms β 102.68 ms β no change β
β QQuery 81 β 27.31 ms β 27.92 ms β no change β
β QQuery 82 β 17.32 ms β 17.64 ms β no change β
β QQuery 83 β 34.56 ms β 35.50 ms β no change β
β QQuery 84 β 30.16 ms β 30.63 ms β no change β
β QQuery 85 β 107.44 ms β 106.73 ms β no change β
β QQuery 86 β 26.71 ms β 26.35 ms β no change β
β QQuery 87 β 67.83 ms β 66.28 ms β no change β
β QQuery 88 β 63.33 ms β 62.64 ms β no change β
β QQuery 89 β 37.03 ms β 36.93 ms β no change β
β QQuery 90 β 17.51 ms β 17.52 ms β no change β
β QQuery 91 β 45.84 ms β 45.51 ms β no change β
β QQuery 92 β 31.48 ms β 31.17 ms β no change β
β QQuery 93 β 51.99 ms β 52.49 ms β no change β
β QQuery 94 β 40.71 ms β 41.41 ms β no change β
β QQuery 95 β 83.91 ms β 83.47 ms β no change β
β QQuery 96 β 24.81 ms β 24.35 ms β no change β
β QQuery 97 β 53.64 ms β 53.03 ms β no change β
β QQuery 98 β 45.44 ms β 45.31 ms β no change β
β QQuery 99 β 68.21 ms β 68.92 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9593.48ms β
β Total Time (spill-dedup-view-arrays) β 9488.96ms β
β Average Time (HEAD) β 96.90ms β
β Average Time (spill-dedup-view-arrays) β 95.85ms β
β Queries Faster β 5 β
β Queries Slower β 1 β
β Queries with No Change β 93 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 6.10 / 6.69 Β±1.03 / 8.74 ms β 6.07 / 6.67 Β±1.01 / 8.67 ms β no change β
β QQuery 2 β 83.35 / 83.68 Β±0.35 / 84.33 ms β 82.53 / 83.07 Β±0.45 / 83.70 ms β no change β
β QQuery 3 β 30.07 / 30.18 Β±0.12 / 30.41 ms β 29.69 / 29.90 Β±0.16 / 30.09 ms β no change β
β QQuery 4 β 507.95 / 512.84 Β±5.14 / 522.06 ms β 491.25 / 504.55 Β±11.84 / 525.82 ms β no change β
β QQuery 5 β 52.03 / 52.73 Β±0.57 / 53.62 ms β 51.85 / 52.22 Β±0.22 / 52.50 ms β no change β
β QQuery 6 β 36.99 / 37.52 Β±0.36 / 37.94 ms β 36.67 / 37.09 Β±0.42 / 37.77 ms β no change β
β QQuery 7 β 75.64 / 76.06 Β±0.26 / 76.42 ms β 75.10 / 75.78 Β±0.42 / 76.21 ms β no change β
β QQuery 8 β 37.38 / 37.86 Β±0.32 / 38.29 ms β 36.92 / 38.86 Β±2.56 / 43.85 ms β no change β
β QQuery 9 β 53.79 / 55.09 Β±0.67 / 55.64 ms β 52.96 / 55.24 Β±1.74 / 58.14 ms β no change β
β QQuery 10 β 62.85 / 63.14 Β±0.32 / 63.66 ms β 62.56 / 62.98 Β±0.32 / 63.40 ms β no change β
β QQuery 11 β 304.08 / 319.08 Β±15.64 / 346.54 ms β 305.38 / 309.31 Β±3.46 / 315.58 ms β no change β
β QQuery 12 β 31.19 / 31.86 Β±0.44 / 32.51 ms β 29.96 / 30.38 Β±0.42 / 31.02 ms β no change β
β QQuery 13 β 124.04 / 125.35 Β±0.75 / 126.24 ms β 121.51 / 123.21 Β±1.50 / 125.58 ms β no change β
β QQuery 14 β 408.01 / 412.01 Β±2.40 / 414.80 ms β 401.59 / 410.59 Β±11.26 / 432.06 ms β no change β
β QQuery 15 β 58.82 / 59.79 Β±0.77 / 60.92 ms β 58.35 / 59.01 Β±0.62 / 59.86 ms β no change β
β QQuery 16 β 6.91 / 7.04 Β±0.10 / 7.21 ms β 6.94 / 7.12 Β±0.10 / 7.20 ms β no change β
β QQuery 17 β 79.90 / 81.59 Β±2.03 / 85.52 ms β 79.89 / 80.94 Β±1.46 / 83.82 ms β no change β
β QQuery 18 β 107.36 / 108.55 Β±0.93 / 109.67 ms β 106.76 / 108.01 Β±0.64 / 108.51 ms β no change β
β QQuery 19 β 42.16 / 42.60 Β±0.28 / 43.02 ms β 41.97 / 42.56 Β±0.34 / 42.87 ms β no change β
β QQuery 20 β 36.82 / 38.14 Β±1.25 / 40.32 ms β 36.79 / 37.10 Β±0.22 / 37.42 ms β no change β
β QQuery 21 β 17.44 / 17.65 Β±0.19 / 17.96 ms β 17.46 / 18.27 Β±1.20 / 20.65 ms β no change β
β QQuery 22 β 66.70 / 67.95 Β±1.42 / 70.56 ms β 67.28 / 68.29 Β±0.88 / 69.37 ms β no change β
β QQuery 23 β 330.10 / 346.18 Β±14.50 / 370.55 ms β 329.26 / 332.39 Β±2.73 / 336.33 ms β no change β
β QQuery 24 β 212.45 / 216.38 Β±3.39 / 220.82 ms β 204.15 / 207.86 Β±4.15 / 215.49 ms β no change β
β QQuery 25 β 112.13 / 113.83 Β±2.15 / 118.08 ms β 109.64 / 111.55 Β±1.98 / 114.54 ms β no change β
β QQuery 26 β 50.14 / 52.03 Β±3.23 / 58.48 ms β 48.76 / 49.13 Β±0.30 / 49.67 ms β +1.06x faster β
β QQuery 27 β 6.85 / 6.94 Β±0.12 / 7.18 ms β 6.44 / 6.61 Β±0.17 / 6.89 ms β no change β
β QQuery 28 β 61.95 / 63.58 Β±1.47 / 66.27 ms β 57.75 / 61.48 Β±3.75 / 67.93 ms β no change β
β QQuery 29 β 98.62 / 100.44 Β±2.17 / 104.49 ms β 96.28 / 98.80 Β±2.85 / 104.22 ms β no change β
β QQuery 30 β 35.10 / 38.42 Β±3.75 / 45.74 ms β 35.16 / 35.70 Β±0.46 / 36.28 ms β +1.08x faster β
β QQuery 31 β 114.96 / 115.93 Β±1.13 / 118.07 ms β 114.77 / 116.50 Β±0.92 / 117.54 ms β no change β
β QQuery 32 β 21.86 / 22.14 Β±0.20 / 22.38 ms β 21.52 / 21.91 Β±0.21 / 22.17 ms β no change β
β QQuery 33 β 39.40 / 39.53 Β±0.14 / 39.71 ms β 38.58 / 38.97 Β±0.31 / 39.41 ms β no change β
β QQuery 34 β 10.70 / 11.45 Β±0.72 / 12.81 ms β 10.44 / 10.65 Β±0.15 / 10.92 ms β +1.07x faster β
β QQuery 35 β 78.32 / 79.63 Β±1.23 / 81.48 ms β 78.25 / 79.73 Β±1.33 / 82.22 ms β no change β
β QQuery 36 β 6.30 / 6.39 Β±0.09 / 6.57 ms β 6.26 / 6.94 Β±1.15 / 9.24 ms β 1.09x slower β
β QQuery 37 β 7.43 / 7.60 Β±0.11 / 7.75 ms β 7.58 / 7.75 Β±0.13 / 7.88 ms β no change β
β QQuery 38 β 65.75 / 66.65 Β±0.60 / 67.65 ms β 67.41 / 67.96 Β±0.67 / 69.21 ms β no change β
β QQuery 39 β 79.39 / 82.09 Β±3.36 / 88.67 ms β 78.22 / 80.58 Β±2.80 / 86.01 ms β no change β
β QQuery 40 β 26.31 / 26.64 Β±0.40 / 27.35 ms β 25.85 / 26.55 Β±0.53 / 27.17 ms β no change β
β QQuery 41 β 12.41 / 12.55 Β±0.10 / 12.71 ms β 12.10 / 12.64 Β±0.81 / 14.24 ms β no change β
β QQuery 42 β 25.12 / 25.39 Β±0.18 / 25.64 ms β 25.24 / 25.80 Β±0.74 / 27.21 ms β no change β
β QQuery 43 β 5.96 / 6.02 Β±0.06 / 6.13 ms β 5.57 / 5.63 Β±0.06 / 5.73 ms β +1.07x faster β
β QQuery 44 β 10.43 / 10.46 Β±0.02 / 10.50 ms β 9.79 / 9.95 Β±0.10 / 10.09 ms β no change β
β QQuery 45 β 47.24 / 48.04 Β±0.57 / 48.80 ms β 45.16 / 46.00 Β±0.88 / 47.67 ms β no change β
β QQuery 46 β 12.78 / 12.93 Β±0.15 / 13.13 ms β 12.36 / 12.83 Β±0.42 / 13.57 ms β no change β
β QQuery 47 β 264.79 / 270.48 Β±6.13 / 280.64 ms β 246.71 / 252.56 Β±8.00 / 268.16 ms β +1.07x faster β
β QQuery 48 β 99.19 / 99.92 Β±0.52 / 100.80 ms β 98.18 / 99.25 Β±0.80 / 100.33 ms β no change β
β QQuery 49 β 73.44 / 74.97 Β±1.23 / 76.67 ms β 72.58 / 73.57 Β±0.84 / 75.01 ms β no change β
β QQuery 50 β 60.34 / 61.19 Β±0.72 / 62.36 ms β 61.54 / 63.80 Β±2.77 / 69.07 ms β no change β
β QQuery 51 β 95.74 / 97.67 Β±1.13 / 99.21 ms β 98.06 / 99.02 Β±1.27 / 101.25 ms β no change β
β QQuery 52 β 25.15 / 25.53 Β±0.42 / 26.32 ms β 25.78 / 25.90 Β±0.15 / 26.20 ms β no change β
β QQuery 53 β 29.91 / 30.69 Β±0.82 / 32.27 ms β 30.42 / 30.56 Β±0.11 / 30.76 ms β no change β
β QQuery 54 β 55.83 / 56.07 Β±0.18 / 56.38 ms β 56.39 / 58.17 Β±2.47 / 63.07 ms β no change β
β QQuery 55 β 24.85 / 25.19 Β±0.38 / 25.86 ms β 24.86 / 25.16 Β±0.44 / 26.02 ms β no change β
β QQuery 56 β 40.39 / 41.12 Β±0.44 / 41.68 ms β 40.59 / 41.47 Β±0.64 / 42.49 ms β no change β
β QQuery 57 β 181.07 / 184.34 Β±3.03 / 188.42 ms β 179.05 / 181.78 Β±3.40 / 188.25 ms β no change β
β QQuery 58 β 114.34 / 116.89 Β±1.86 / 119.19 ms β 113.42 / 115.69 Β±2.24 / 119.98 ms β no change β
β QQuery 59 β 119.87 / 120.35 Β±0.34 / 120.75 ms β 119.29 / 120.47 Β±1.09 / 122.42 ms β no change β
β QQuery 60 β 40.79 / 42.28 Β±1.48 / 45.08 ms β 40.42 / 40.97 Β±0.50 / 41.74 ms β no change β
β QQuery 61 β 12.25 / 12.48 Β±0.20 / 12.84 ms β 12.31 / 12.49 Β±0.12 / 12.66 ms β no change β
β QQuery 62 β 45.67 / 46.19 Β±0.40 / 46.90 ms β 45.39 / 47.14 Β±2.38 / 51.87 ms β no change β
β QQuery 63 β 30.83 / 31.34 Β±0.38 / 31.82 ms β 30.17 / 31.32 Β±1.08 / 33.36 ms β no change β
β QQuery 64 β 374.19 / 383.45 Β±10.29 / 403.21 ms β 368.65 / 375.45 Β±4.97 / 383.51 ms β no change β
β QQuery 65 β 134.35 / 135.92 Β±2.11 / 140.10 ms β 134.52 / 136.36 Β±1.61 / 139.34 ms β no change β
β QQuery 66 β 80.02 / 80.97 Β±1.02 / 82.90 ms β 79.74 / 83.53 Β±4.39 / 91.66 ms β no change β
β QQuery 67 β 263.33 / 271.56 Β±5.36 / 278.25 ms β 267.33 / 271.34 Β±3.83 / 278.20 ms β no change β
β QQuery 68 β 12.72 / 12.83 Β±0.08 / 12.95 ms β 12.17 / 12.39 Β±0.11 / 12.48 ms β no change β
β QQuery 69 β 58.56 / 62.01 Β±5.24 / 72.45 ms β 56.96 / 57.58 Β±0.41 / 58.05 ms β +1.08x faster β
β QQuery 70 β 107.62 / 109.66 Β±2.20 / 113.86 ms β 104.98 / 106.59 Β±0.98 / 107.65 ms β no change β
β QQuery 71 β 36.22 / 38.35 Β±3.54 / 45.39 ms β 35.99 / 36.90 Β±1.49 / 39.87 ms β no change β
β QQuery 72 β 1859.33 / 1904.58 Β±43.85 / 1987.28 ms β 1817.92 / 1939.79 Β±74.31 / 2026.30 ms β no change β
β QQuery 73 β 10.47 / 10.92 Β±0.55 / 11.94 ms β 10.88 / 11.37 Β±0.40 / 12.00 ms β no change β
β QQuery 74 β 176.46 / 181.24 Β±3.43 / 184.78 ms β 190.90 / 194.78 Β±4.31 / 203.08 ms β 1.07x slower β
β QQuery 75 β 144.20 / 145.53 Β±1.80 / 148.72 ms β 147.67 / 149.56 Β±1.45 / 151.40 ms β no change β
β QQuery 76 β 35.17 / 35.41 Β±0.22 / 35.75 ms β 36.34 / 37.65 Β±1.22 / 39.86 ms β 1.06x slower β
β QQuery 77 β 62.91 / 63.45 Β±0.51 / 64.30 ms β 64.03 / 64.36 Β±0.30 / 64.85 ms β no change β
β QQuery 78 β 177.07 / 180.25 Β±3.79 / 187.38 ms β 173.90 / 178.02 Β±3.76 / 184.41 ms β no change β
β QQuery 79 β 70.15 / 70.66 Β±0.71 / 72.05 ms β 69.34 / 70.77 Β±0.90 / 72.04 ms β no change β
β QQuery 80 β 99.72 / 102.98 Β±3.60 / 109.79 ms β 102.68 / 103.76 Β±0.92 / 105.23 ms β no change β
β QQuery 81 β 27.31 / 27.50 Β±0.19 / 27.86 ms β 27.92 / 28.14 Β±0.19 / 28.46 ms β no change β
β QQuery 82 β 17.32 / 17.50 Β±0.12 / 17.65 ms β 17.64 / 17.75 Β±0.06 / 17.82 ms β no change β
β QQuery 83 β 34.56 / 35.08 Β±0.30 / 35.39 ms β 35.50 / 35.78 Β±0.31 / 36.18 ms β no change β
β QQuery 84 β 30.16 / 32.36 Β±3.33 / 38.96 ms β 30.63 / 30.81 Β±0.22 / 31.21 ms β no change β
β QQuery 85 β 107.44 / 109.01 Β±1.24 / 110.62 ms β 106.73 / 110.65 Β±3.50 / 116.98 ms β no change β
β QQuery 86 β 26.71 / 27.32 Β±0.36 / 27.77 ms β 26.35 / 26.83 Β±0.28 / 27.22 ms β no change β
β QQuery 87 β 67.83 / 70.24 Β±3.32 / 76.83 ms β 66.28 / 66.86 Β±0.35 / 67.33 ms β no change β
β QQuery 88 β 63.33 / 64.25 Β±0.79 / 65.61 ms β 62.64 / 64.40 Β±2.59 / 69.50 ms β no change β
β QQuery 89 β 37.03 / 37.55 Β±0.48 / 38.35 ms β 36.93 / 37.31 Β±0.36 / 37.93 ms β no change β
β QQuery 90 β 17.51 / 17.65 Β±0.18 / 18.00 ms β 17.52 / 17.66 Β±0.11 / 17.84 ms β no change β
β QQuery 91 β 45.84 / 46.10 Β±0.15 / 46.29 ms β 45.51 / 46.17 Β±0.35 / 46.45 ms β no change β
β QQuery 92 β 31.48 / 33.86 Β±3.75 / 41.30 ms β 31.17 / 31.74 Β±0.34 / 32.18 ms β +1.07x faster β
β QQuery 93 β 51.99 / 52.86 Β±0.54 / 53.42 ms β 52.49 / 52.87 Β±0.38 / 53.38 ms β no change β
β QQuery 94 β 40.71 / 41.31 Β±0.50 / 41.96 ms β 41.41 / 42.20 Β±0.72 / 43.55 ms β no change β
β QQuery 95 β 83.91 / 84.81 Β±0.66 / 85.89 ms β 83.47 / 84.14 Β±0.54 / 84.86 ms β no change β
β QQuery 96 β 24.81 / 26.47 Β±3.20 / 32.86 ms β 24.35 / 24.51 Β±0.14 / 24.73 ms β +1.08x faster β
β QQuery 97 β 53.64 / 55.22 Β±2.21 / 59.59 ms β 53.03 / 56.02 Β±4.01 / 63.96 ms β no change β
β QQuery 98 β 45.44 / 45.90 Β±0.52 / 46.67 ms β 45.31 / 46.40 Β±1.05 / 48.23 ms β no change β
β QQuery 99 β 68.21 / 68.74 Β±0.33 / 69.08 ms β 68.92 / 69.47 Β±0.52 / 70.14 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9798.80ms β
β Total Time (spill-dedup-view-arrays) β 9754.24ms β
β Average Time (HEAD) β 98.98ms β
β Average Time (spill-dedup-view-arrays) β 98.53ms β
β Queries Faster β 8 β
β Queries Slower β 3 β
β Queries with No Change β 88 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.0 GiB |
| Avg memory | 1.6 GiB |
| CPU user | 211.9s |
| CPU sys | 6.4s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.6 GiB |
| CPU user | 210.9s |
| CPU sys | 6.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783828579
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783725982-2602-wh244 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783903325
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5215412029
> Those benchmarks don't actually stress this path at all. In fact I don't think there are any standard benchmarks that will sweat this particular change. There is a `sort_tpch` benchmark but that has a high cardinality `l_comment` string view column, which, once again, doesn't hit this path or show any benefits here.
we can compare the repeated-label reproduction with a forced-spill high-cardinality case such as sort-TPCH Q3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863111096
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
baseline:
ref: ed43e69fdca8737c9d065b53b5fc5e4183302775
changed:
ref: ed43e69fdca8737c9d065b53b5fc5e4183302775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870646171
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5870585310-2879-wkv4j 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969768746
Those benchmarks don't actually stress this path at all. In fact I don't think there are any standard benchmarks that will sweat this particular change. There is a `sort_tpch` benchmark but that has a high cardinality `l_comment` string view column, which, once again, doesn't hit this path or show any benefits here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863110116
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863110591
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863469607
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110591)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.26 105.0Β±1.11ms ? ?/sec 1.00 83.5Β±0.27ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.09 101.6Β±0.51ms ? ?/sec 1.00 93.0Β±0.26ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.06 99.2Β±0.74ms ? ?/sec 1.00 93.3Β±0.62ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 128.1Β±1.97ms ? ?/sec 1.00 128.4Β±1.51ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 111.1Β±0.93ms ? ?/sec 1.01 111.7Β±0.72ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 138.8 MiB | 138.8 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 120.6 MiB | 120.6 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69 (merge-base)`) | 138.8 MiB | 836.8 MiB | 698.1 MiB | 6.0Γ |
| spill_views | changed (`spill-dedup-view-arrays`) | 138.8 MiB | 811.2 MiB | 672.5 MiB | 5.8Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 660.2s |
| Peak memory | 836.8 MiB |
| Avg memory | 34.1 MiB |
| CPU user | 69.2s |
| CPU sys | 29.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 660.2s |
| Peak memory | 811.2 MiB |
| Avg memory | 39.0 MiB |
| CPU user | 82.4s |
| CPU sys | 21.1s |
| Peak spill | 80.0 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863248341
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111696)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863111696-2863-mnqnf 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783313493
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.79 ms β 5.57 ms β no change β
β QQuery 2 β 81.91 ms β 81.02 ms β no change β
β QQuery 3 β 29.18 ms β 28.90 ms β no change β
β QQuery 4 β 504.90 ms β 457.95 ms β +1.10x faster β
β QQuery 5 β 53.11 ms β 51.82 ms β no change β
β QQuery 6 β 38.47 ms β 35.94 ms β +1.07x faster β
β QQuery 7 β 76.73 ms β 75.20 ms β no change β
β QQuery 8 β 37.50 ms β 36.77 ms β no change β
β QQuery 9 β 53.21 ms β 53.34 ms β no change β
β QQuery 10 β 62.63 ms β 62.58 ms β no change β
β QQuery 11 β 295.40 ms β 301.89 ms β no change β
β QQuery 12 β 30.39 ms β 29.23 ms β no change β
β QQuery 13 β 121.28 ms β 118.93 ms β no change β
β QQuery 14 β 398.55 ms β 398.77 ms β no change β
β QQuery 15 β 63.12 ms β 58.42 ms β +1.08x faster β
β QQuery 16 β 7.00 ms β 6.77 ms β no change β
β QQuery 17 β 81.24 ms β 78.81 ms β no change β
β QQuery 18 β 104.94 ms β 106.83 ms β no change β
β QQuery 19 β 41.90 ms β 41.94 ms β no change β
β QQuery 20 β 36.46 ms β 35.98 ms β no change β
β QQuery 21 β 17.58 ms β 17.19 ms β no change β
β QQuery 22 β 66.97 ms β 66.55 ms β no change β
β QQuery 23 β 323.54 ms β 313.00 ms β no change β
β QQuery 24 β 203.08 ms β 194.80 ms β no change β
β QQuery 25 β 108.16 ms β 107.62 ms β no change β
β QQuery 26 β 48.49 ms β 48.85 ms β no change β
β QQuery 27 β 6.24 ms β 6.36 ms β no change β
β QQuery 28 β 56.57 ms β 60.68 ms β 1.07x slower β
β QQuery 29 β 95.86 ms β 95.75 ms β no change β
β QQuery 30 β 34.08 ms β 33.29 ms β no change β
β QQuery 31 β 111.16 ms β 110.85 ms β no change β
β QQuery 32 β 21.24 ms β 21.14 ms β no change β
β QQuery 33 β 38.28 ms β 37.70 ms β no change β
β QQuery 34 β 10.65 ms β 10.28 ms β no change β
β QQuery 35 β 74.49 ms β 72.22 ms β no change β
β QQuery 36 β 6.16 ms β 5.64 ms β +1.09x faster β
β QQuery 37 β 7.24 ms β 6.98 ms β no change β
β QQuery 38 β 63.99 ms β 62.59 ms β no change β
β QQuery 39 β 75.42 ms β 73.93 ms β no change β
β QQuery 40 β 24.92 ms β 23.61 ms β +1.06x faster β
β QQuery 41 β 11.66 ms β 11.38 ms β no change β
β QQuery 42 β 24.21 ms β 24.11 ms β no change β
β QQuery 43 β 5.41 ms β 5.14 ms β no change β
β QQuery 44 β 9.49 ms β 9.44 ms β no change β
β QQuery 45 β 40.52 ms β 38.28 ms β +1.06x faster β
β QQuery 46 β 12.22 ms β 11.62 ms β no change β
β QQuery 47 β 226.60 ms β 219.15 ms β no change β
β QQuery 48 β 95.90 ms β 95.51 ms β no change β
β QQuery 49 β 70.60 ms β 69.87 ms β no change β
β QQuery 50 β 57.57 ms β 58.10 ms β no change β
β QQuery 51 β 92.20 ms β 91.58 ms β no change β
β QQuery 52 β 24.32 ms β 24.92 ms β no change β
β QQuery 53 β 29.00 ms β 29.70 ms β no change β
β QQuery 54 β 54.24 ms β 55.33 ms β no change β
β QQuery 55 β 23.71 ms β 24.11 ms β no change β
β QQuery 56 β 38.84 ms β 39.35 ms β no change β
β QQuery 57 β 169.71 ms β 169.10 ms β no change β
β QQuery 58 β 111.80 ms β 111.64 ms β no change β
β QQuery 59 β 117.48 ms β 116.59 ms β no change β
β QQuery 60 β 39.30 ms β 39.47 ms β no change β
β QQuery 61 β 11.91 ms β 12.11 ms β no change β
β QQuery 62 β 45.12 ms β 45.24 ms β no change β
β QQuery 63 β 29.31 ms β 29.68 ms β no change β
β QQuery 64 β 356.79 ms β 368.93 ms β no change β
β QQuery 65 β 133.23 ms β 133.51 ms β no change β
β QQuery 66 β 79.60 ms β 79.15 ms β no change β
β QQuery 67 β 262.48 ms β 258.93 ms β no change β
β QQuery 68 β 12.27 ms β 12.28 ms β no change β
β QQuery 69 β 57.42 ms β 57.49 ms β no change β
β QQuery 70 β 105.53 ms β 105.35 ms β no change β
β QQuery 71 β 35.70 ms β 35.60 ms β no change β
β QQuery 72 β 1806.74 ms β 1817.04 ms β no change β
β QQuery 73 β 10.32 ms β 10.09 ms β no change β
β QQuery 74 β 166.42 ms β 159.97 ms β no change β
β QQuery 75 β 145.01 ms β 143.47 ms β no change β
β QQuery 76 β 35.43 ms β 35.16 ms β no change β
β QQuery 77 β 61.96 ms β 61.53 ms β no change β
β QQuery 78 β 164.04 ms β 165.43 ms β no change β
β QQuery 79 β 67.13 ms β 66.21 ms β no change β
β QQuery 80 β 97.94 ms β 95.89 ms β no change β
β QQuery 81 β 26.70 ms β 27.09 ms β no change β
β QQuery 82 β 17.10 ms β 17.20 ms β no change β
β QQuery 83 β 33.97 ms β 35.01 ms β no change β
β QQuery 84 β 30.01 ms β 30.04 ms β no change β
β QQuery 85 β 106.33 ms β 105.09 ms β no change β
β QQuery 86 β 26.68 ms β 26.37 ms β no change β
β QQuery 87 β 64.62 ms β 64.27 ms β no change β
β QQuery 88 β 62.11 ms β 61.66 ms β no change β
β QQuery 89 β 36.21 ms β 35.18 ms β no change β
β QQuery 90 β 17.14 ms β 17.26 ms β no change β
β QQuery 91 β 45.16 ms β 45.22 ms β no change β
β QQuery 92 β 30.29 ms β 31.06 ms β no change β
β QQuery 93 β 51.17 ms β 50.46 ms β no change β
β QQuery 94 β 39.87 ms β 40.47 ms β no change β
β QQuery 95 β 81.40 ms β 82.43 ms β no change β
β QQuery 96 β 24.23 ms β 23.99 ms β no change β
β QQuery 97 β 51.32 ms β 52.12 ms β no change β
β QQuery 98 β 43.25 ms β 43.74 ms β no change β
β QQuery 99 β 66.66 ms β 67.66 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9311.19ms β
β Total Time (spill-dedup-view-arrays) β 9229.47ms β
β Average Time (HEAD) β 94.05ms β
β Average Time (spill-dedup-view-arrays) β 93.23ms β
β Queries Faster β 6 β
β Queries Slower β 1 β
β Queries with No Change β 92 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.79 / 6.32 Β±0.97 / 8.26 ms β 5.57 / 6.18 Β±1.00 / 8.16 ms β no change β
β QQuery 2 β 81.91 / 82.75 Β±0.71 / 83.74 ms β 81.02 / 81.15 Β±0.14 / 81.38 ms β no change β
β QQuery 3 β 29.18 / 29.47 Β±0.21 / 29.73 ms β 28.90 / 29.17 Β±0.15 / 29.33 ms β no change β
β QQuery 4 β 504.90 / 513.43 Β±7.09 / 526.25 ms β 457.95 / 479.07 Β±18.71 / 506.92 ms β +1.07x faster β
β QQuery 5 β 53.11 / 53.88 Β±0.64 / 54.82 ms β 51.82 / 52.92 Β±0.81 / 54.22 ms β no change β
β QQuery 6 β 38.47 / 39.50 Β±0.57 / 40.15 ms β 35.94 / 36.92 Β±0.56 / 37.50 ms β +1.07x faster β
β QQuery 7 β 76.73 / 77.04 Β±0.18 / 77.23 ms β 75.20 / 76.15 Β±0.60 / 76.99 ms β no change β
β QQuery 8 β 37.50 / 39.58 Β±2.35 / 44.13 ms β 36.77 / 36.98 Β±0.21 / 37.30 ms β +1.07x faster β
β QQuery 9 β 53.21 / 55.54 Β±1.70 / 58.29 ms β 53.34 / 55.18 Β±1.55 / 57.07 ms β no change β
β QQuery 10 β 62.63 / 63.31 Β±0.50 / 64.03 ms β 62.58 / 63.29 Β±0.48 / 64.05 ms β no change β
β QQuery 11 β 295.40 / 307.26 Β±6.58 / 314.08 ms β 301.89 / 305.90 Β±4.10 / 312.26 ms β no change β
β QQuery 12 β 30.39 / 30.91 Β±0.39 / 31.52 ms β 29.23 / 29.77 Β±0.38 / 30.34 ms β no change β
β QQuery 13 β 121.28 / 123.30 Β±1.29 / 125.01 ms β 118.93 / 119.97 Β±0.94 / 121.25 ms β no change β
β QQuery 14 β 398.55 / 404.13 Β±4.35 / 411.14 ms β 398.77 / 403.51 Β±3.09 / 407.80 ms β no change β
β QQuery 15 β 63.12 / 63.83 Β±0.49 / 64.42 ms β 58.42 / 59.32 Β±0.97 / 61.15 ms β +1.08x faster β
β QQuery 16 β 7.00 / 7.28 Β±0.24 / 7.69 ms β 6.77 / 6.84 Β±0.05 / 6.91 ms β +1.06x faster β
β QQuery 17 β 81.24 / 82.17 Β±1.19 / 84.46 ms β 78.81 / 80.68 Β±1.49 / 82.59 ms β no change β
β QQuery 18 β 104.94 / 106.70 Β±1.21 / 108.06 ms β 106.83 / 109.89 Β±3.70 / 117.17 ms β no change β
β QQuery 19 β 41.90 / 42.40 Β±0.38 / 42.90 ms β 41.94 / 42.35 Β±0.36 / 42.92 ms β no change β
β QQuery 20 β 36.46 / 36.79 Β±0.33 / 37.35 ms β 35.98 / 36.30 Β±0.28 / 36.79 ms β no change β
β QQuery 21 β 17.58 / 18.14 Β±0.74 / 19.58 ms β 17.19 / 18.18 Β±1.51 / 21.16 ms β no change β
β QQuery 22 β 66.97 / 67.76 Β±0.89 / 69.22 ms β 66.55 / 66.83 Β±0.25 / 67.16 ms β no change β
β QQuery 23 β 323.54 / 327.82 Β±2.78 / 332.06 ms β 313.00 / 322.49 Β±7.56 / 334.48 ms β no change β
β QQuery 24 β 203.08 / 207.67 Β±5.50 / 218.03 ms β 194.80 / 199.82 Β±5.56 / 210.52 ms β no change β
β QQuery 25 β 108.16 / 110.55 Β±1.82 / 113.68 ms β 107.62 / 109.57 Β±1.15 / 110.99 ms β no change β
β QQuery 26 β 48.49 / 48.75 Β±0.19 / 48.98 ms β 48.85 / 49.19 Β±0.28 / 49.52 ms β no change β
β QQuery 27 β 6.24 / 6.37 Β±0.16 / 6.68 ms β 6.36 / 6.58 Β±0.16 / 6.75 ms β no change β
β QQuery 28 β 56.57 / 61.44 Β±3.41 / 66.23 ms β 60.68 / 62.39 Β±1.42 / 64.95 ms β no change β
β QQuery 29 β 95.86 / 98.12 Β±2.24 / 102.21 ms β 95.75 / 97.88 Β±3.10 / 104.01 ms β no change β
β QQuery 30 β 34.08 / 34.20 Β±0.11 / 34.38 ms β 33.29 / 33.55 Β±0.22 / 33.87 ms β no change β
β QQuery 31 β 111.16 / 113.34 Β±3.28 / 119.83 ms β 110.85 / 112.46 Β±1.13 / 113.97 ms β no change β
β QQuery 32 β 21.24 / 21.56 Β±0.19 / 21.79 ms β 21.14 / 21.39 Β±0.19 / 21.65 ms β no change β
β QQuery 33 β 38.28 / 38.62 Β±0.22 / 38.89 ms β 37.70 / 38.94 Β±1.63 / 42.09 ms β no change β
β QQuery 34 β 10.65 / 10.92 Β±0.42 / 11.74 ms β 10.28 / 10.52 Β±0.18 / 10.79 ms β no change β
β QQuery 35 β 74.49 / 76.60 Β±2.50 / 81.51 ms β 72.22 / 73.60 Β±1.75 / 77.04 ms β no change β
β QQuery 36 β 6.16 / 6.87 Β±0.94 / 8.71 ms β 5.64 / 5.79 Β±0.19 / 6.15 ms β +1.19x faster β
β QQuery 37 β 7.24 / 7.44 Β±0.20 / 7.82 ms β 6.98 / 7.22 Β±0.13 / 7.37 ms β no change β
β QQuery 38 β 63.99 / 64.92 Β±0.91 / 66.66 ms β 62.59 / 63.10 Β±0.56 / 64.03 ms β no change β
β QQuery 39 β 75.42 / 75.95 Β±0.47 / 76.63 ms β 73.93 / 74.66 Β±0.57 / 75.56 ms β no change β
β QQuery 40 β 24.92 / 26.45 Β±2.34 / 31.10 ms β 23.61 / 23.78 Β±0.12 / 23.91 ms β +1.11x faster β
β QQuery 41 β 11.66 / 11.76 Β±0.11 / 11.96 ms β 11.38 / 11.59 Β±0.13 / 11.79 ms β no change β
β QQuery 42 β 24.21 / 25.12 Β±0.73 / 26.23 ms β 24.11 / 24.73 Β±0.88 / 26.48 ms β no change β
β QQuery 43 β 5.41 / 5.54 Β±0.14 / 5.79 ms β 5.14 / 5.29 Β±0.15 / 5.59 ms β no change β
β QQuery 44 β 9.49 / 9.63 Β±0.11 / 9.75 ms β 9.44 / 9.50 Β±0.05 / 9.57 ms β no change β
β QQuery 45 β 40.52 / 41.95 Β±0.82 / 42.89 ms β 38.28 / 39.22 Β±0.63 / 40.22 ms β +1.07x faster β
β QQuery 46 β 12.22 / 12.41 Β±0.22 / 12.84 ms β 11.62 / 11.79 Β±0.13 / 12.01 ms β +1.05x faster β
β QQuery 47 β 226.60 / 234.35 Β±6.50 / 241.87 ms β 219.15 / 221.84 Β±2.40 / 226.04 ms β +1.06x faster β
β QQuery 48 β 95.90 / 97.91 Β±1.96 / 101.20 ms β 95.51 / 95.93 Β±0.35 / 96.53 ms β no change β
β QQuery 49 β 70.60 / 70.93 Β±0.47 / 71.85 ms β 69.87 / 70.11 Β±0.16 / 70.30 ms β no change β
β QQuery 50 β 57.57 / 57.88 Β±0.22 / 58.13 ms β 58.10 / 58.68 Β±0.70 / 60.02 ms β no change β
β QQuery 51 β 92.20 / 97.67 Β±5.28 / 107.00 ms β 91.58 / 96.92 Β±5.81 / 108.09 ms β no change β
β QQuery 52 β 24.32 / 24.78 Β±0.52 / 25.72 ms β 24.92 / 25.32 Β±0.45 / 26.15 ms β no change β
β QQuery 53 β 29.00 / 29.27 Β±0.20 / 29.58 ms β 29.70 / 30.04 Β±0.18 / 30.20 ms β no change β
β QQuery 54 β 54.24 / 54.57 Β±0.20 / 54.84 ms β 55.33 / 55.85 Β±0.33 / 56.24 ms β no change β
β QQuery 55 β 23.71 / 26.20 Β±4.08 / 34.30 ms β 24.11 / 24.27 Β±0.16 / 24.56 ms β +1.08x faster β
β QQuery 56 β 38.84 / 39.56 Β±0.56 / 40.41 ms β 39.35 / 39.53 Β±0.12 / 39.73 ms β no change β
β QQuery 57 β 169.71 / 172.02 Β±3.83 / 179.62 ms β 169.10 / 173.27 Β±4.91 / 182.15 ms β no change β
β QQuery 58 β 111.80 / 112.63 Β±0.91 / 114.22 ms β 111.64 / 113.58 Β±1.87 / 117.12 ms β no change β
β QQuery 59 β 117.48 / 119.44 Β±2.82 / 124.96 ms β 116.59 / 117.48 Β±0.68 / 118.51 ms β no change β
β QQuery 60 β 39.30 / 40.22 Β±0.89 / 41.56 ms β 39.47 / 40.66 Β±1.11 / 42.75 ms β no change β
β QQuery 61 β 11.91 / 11.99 Β±0.05 / 12.04 ms β 12.11 / 12.29 Β±0.15 / 12.56 ms β no change β
β QQuery 62 β 45.12 / 45.52 Β±0.28 / 45.96 ms β 45.24 / 45.62 Β±0.32 / 46.05 ms β no change β
β QQuery 63 β 29.31 / 29.56 Β±0.16 / 29.83 ms β 29.68 / 29.98 Β±0.20 / 30.32 ms β no change β
β QQuery 64 β 356.79 / 362.33 Β±6.12 / 374.20 ms β 368.93 / 373.34 Β±4.20 / 381.26 ms β no change β
β QQuery 65 β 133.23 / 136.49 Β±2.40 / 140.34 ms β 133.51 / 136.76 Β±3.14 / 142.15 ms β no change β
β QQuery 66 β 79.60 / 80.40 Β±0.67 / 81.59 ms β 79.15 / 79.99 Β±0.59 / 80.78 ms β no change β
β QQuery 67 β 262.48 / 268.15 Β±5.00 / 277.35 ms β 258.93 / 264.67 Β±5.16 / 271.75 ms β no change β
β QQuery 68 β 12.27 / 12.38 Β±0.18 / 12.74 ms β 12.28 / 12.43 Β±0.10 / 12.55 ms β no change β
β QQuery 69 β 57.42 / 57.94 Β±0.46 / 58.63 ms β 57.49 / 59.72 Β±4.02 / 67.76 ms β no change β
β QQuery 70 β 105.53 / 106.66 Β±0.67 / 107.56 ms β 105.35 / 105.95 Β±0.66 / 107.17 ms β no change β
β QQuery 71 β 35.70 / 37.78 Β±2.96 / 43.49 ms β 35.60 / 35.83 Β±0.18 / 36.03 ms β +1.05x faster β
β QQuery 72 β 1806.74 / 1886.19 Β±78.73 / 2015.12 ms β 1817.04 / 1840.88 Β±23.32 / 1875.93 ms β no change β
β QQuery 73 β 10.32 / 10.60 Β±0.18 / 10.85 ms β 10.09 / 10.87 Β±0.75 / 12.08 ms β no change β
β QQuery 74 β 166.42 / 172.65 Β±6.17 / 182.97 ms β 159.97 / 161.44 Β±1.65 / 164.64 ms β +1.07x faster β
β QQuery 75 β 145.01 / 145.94 Β±0.56 / 146.56 ms β 143.47 / 146.69 Β±2.77 / 151.65 ms β no change β
β QQuery 76 β 35.43 / 35.69 Β±0.32 / 36.29 ms β 35.16 / 35.64 Β±0.50 / 36.57 ms β no change β
β QQuery 77 β 61.96 / 62.29 Β±0.27 / 62.67 ms β 61.53 / 64.55 Β±2.63 / 68.02 ms β no change β
β QQuery 78 β 164.04 / 168.28 Β±2.51 / 171.75 ms β 165.43 / 170.53 Β±4.30 / 177.82 ms β no change β
β QQuery 79 β 67.13 / 67.71 Β±0.37 / 68.14 ms β 66.21 / 68.24 Β±2.30 / 72.70 ms β no change β
β QQuery 80 β 97.94 / 99.84 Β±1.78 / 102.74 ms β 95.89 / 97.76 Β±1.15 / 99.08 ms β no change β
β QQuery 81 β 26.70 / 27.24 Β±0.34 / 27.73 ms β 27.09 / 28.11 Β±1.80 / 31.70 ms β no change β
β QQuery 82 β 17.10 / 17.22 Β±0.10 / 17.33 ms β 17.20 / 18.27 Β±1.11 / 20.41 ms β 1.06x slower β
β QQuery 83 β 33.97 / 34.46 Β±0.27 / 34.71 ms β 35.01 / 36.34 Β±2.55 / 41.44 ms β 1.05x slower β
β QQuery 84 β 30.01 / 30.17 Β±0.11 / 30.34 ms β 30.04 / 30.77 Β±0.70 / 31.92 ms β no change β
β QQuery 85 β 106.33 / 109.00 Β±4.13 / 117.23 ms β 105.09 / 105.54 Β±0.46 / 106.40 ms β no change β
β QQuery 86 β 26.68 / 26.99 Β±0.31 / 27.58 ms β 26.37 / 27.45 Β±1.11 / 29.31 ms β no change β
β QQuery 87 β 64.62 / 65.15 Β±0.32 / 65.53 ms β 64.27 / 66.92 Β±3.53 / 73.88 ms β no change β
β QQuery 88 β 62.11 / 64.14 Β±2.72 / 69.38 ms β 61.66 / 61.97 Β±0.28 / 62.31 ms β no change β
β QQuery 89 β 36.21 / 36.71 Β±0.31 / 37.11 ms β 35.18 / 35.41 Β±0.12 / 35.51 ms β no change β
β QQuery 90 β 17.14 / 17.39 Β±0.17 / 17.66 ms β 17.26 / 17.43 Β±0.14 / 17.61 ms β no change β
β QQuery 91 β 45.16 / 45.58 Β±0.29 / 45.96 ms β 45.22 / 47.00 Β±2.14 / 51.21 ms β no change β
β QQuery 92 β 30.29 / 30.85 Β±0.62 / 31.98 ms β 31.06 / 31.51 Β±0.47 / 32.28 ms β no change β
β QQuery 93 β 51.17 / 52.27 Β±1.64 / 55.53 ms β 50.46 / 51.40 Β±0.82 / 52.55 ms β no change β
β QQuery 94 β 39.87 / 40.59 Β±0.90 / 42.30 ms β 40.47 / 40.95 Β±0.30 / 41.31 ms β no change β
β QQuery 95 β 81.40 / 81.93 Β±0.47 / 82.79 ms β 82.43 / 83.69 Β±1.03 / 85.33 ms β no change β
β QQuery 96 β 24.23 / 24.46 Β±0.25 / 24.86 ms β 23.99 / 24.58 Β±0.45 / 25.20 ms β no change β
β QQuery 97 β 51.32 / 53.63 Β±2.44 / 57.94 ms β 52.12 / 54.17 Β±1.52 / 55.74 ms β no change β
β QQuery 98 β 43.25 / 43.84 Β±0.44 / 44.60 ms β 43.74 / 44.57 Β±1.09 / 46.72 ms β no change β
β QQuery 99 β 66.66 / 67.20 Β±0.43 / 67.92 ms β 67.66 / 70.82 Β±2.63 / 74.41 ms β 1.05x slower β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9540.11ms β
β Total Time (spill-dedup-view-arrays) β 9405.15ms β
β Average Time (HEAD) β 96.36ms β
β Average Time (spill-dedup-view-arrays) β 95.00ms β
β Queries Faster β 13 β
β Queries Slower β 3 β
β Queries with No Change β 83 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.5 GiB |
| CPU user | 205.5s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.5 GiB |
| Avg memory | 1.7 GiB |
| CPU user | 203.8s |
| CPU sys | 5.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783733220
run benchmark spill_views
env:
QUERY: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784073068
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783906160)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783906160-2612-dw9mj 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784306558
run benchmark spill_views
env:
BENCH_QUERY: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783904761
run benchmark spill_views
env:
QUERY: "5"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784541902
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784305266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q04_sort_string_all_distinct_distinct 1.00 123.5Β±1.38ms ? ?/sec 1.45 178.6Β±1.91ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 330.2s |
| Peak memory | 378.7 MiB |
| Avg memory | 15.3 MiB |
| CPU user | 11.9s |
| CPU sys | 5.8s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 335.2s |
| Peak memory | 431.8 MiB |
| Avg memory | 15.2 MiB |
| CPU user | 11.1s |
| CPU sys | 4.8s |
| Peak spill | 82.1 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789124786
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789123708
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #23565:
URL: https://github.com/apache/datafusion/pull/23565
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969075084
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4969050947-1049-jv4c6 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (25cc81b71537b63a872d994c8b0512b0bb715fa0) to f755cb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/f755cb44fa33c69837a8774be862fbfe01e7a100..25cc81b71537b63a872d994c8b0512b0bb715fa0) using: tpch
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248884163
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23565?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.98%. Comparing base ([`a9b61ab`](https://app.codecov.io/gh/apache/datafusion/commit/a9b61abf04b59ff08e7254b598b07f4fa058c5d1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`20bc3e1`](https://app.codecov.io/gh/apache/datafusion/commit/20bc3e1a140f8519ce80af05a8d446de9a505d8a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23565 +/- ##
==========================================
- Coverage 80.99% 80.98% -0.01%
==========================================
Files 1106 1106
Lines 383549 383568 +19
Branches 383549 383568 +19
==========================================
+ Hits 310644 310651 +7
- Misses 54570 54578 +8
- Partials 18335 18339 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23565?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783864112
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.28 ms β 1.28 ms β no change β
β QQuery 1 β 12.25 ms β 12.01 ms β no change β
β QQuery 2 β 37.34 ms β 36.83 ms β no change β
β QQuery 3 β 33.04 ms β 31.59 ms β no change β
β QQuery 4 β 252.42 ms β 250.67 ms β no change β
β QQuery 5 β 281.67 ms β 282.29 ms β no change β
β QQuery 6 β 1.28 ms β 1.34 ms β no change β
β QQuery 7 β 12.81 ms β 13.24 ms β no change β
β QQuery 8 β 347.37 ms β 353.59 ms β no change β
β QQuery 9 β 489.29 ms β 488.81 ms β no change β
β QQuery 10 β 65.19 ms β 66.13 ms β no change β
β QQuery 11 β 75.86 ms β 76.45 ms β no change β
β QQuery 12 β 267.11 ms β 277.32 ms β no change β
β QQuery 13 β 385.73 ms β 376.75 ms β no change β
β QQuery 14 β 280.92 ms β 294.05 ms β no change β
β QQuery 15 β 289.54 ms β 299.84 ms β no change β
β QQuery 16 β 640.17 ms β 644.58 ms β no change β
β QQuery 17 β 659.43 ms β 653.97 ms β no change β
β QQuery 18 β 1359.44 ms β 1315.20 ms β no change β
β QQuery 19 β 28.96 ms β 27.44 ms β +1.06x faster β
β QQuery 20 β 521.31 ms β 527.01 ms β no change β
β QQuery 21 β 522.04 ms β 537.00 ms β no change β
β QQuery 22 β 1013.84 ms β 1024.78 ms β no change β
β QQuery 23 β 3111.93 ms β 3190.84 ms β no change β
β QQuery 24 β 41.91 ms β 40.44 ms β no change β
β QQuery 25 β 104.85 ms β 107.94 ms β no change β
β QQuery 26 β 41.94 ms β 42.11 ms β no change β
β QQuery 27 β 524.08 ms β 528.92 ms β no change β
β QQuery 28 β 2965.01 ms β 2969.80 ms β no change β
β QQuery 29 β 42.48 ms β 42.46 ms β no change β
β QQuery 30 β 313.42 ms β 328.08 ms β no change β
β QQuery 31 β 302.02 ms β 290.50 ms β no change β
β QQuery 32 β 1136.10 ms β 1127.80 ms β no change β
β QQuery 33 β 1605.93 ms β 1695.52 ms β 1.06x slower β
β QQuery 34 β 1646.71 ms β 1667.15 ms β no change β
β QQuery 35 β 319.39 ms β 325.07 ms β no change β
β QQuery 36 β 68.88 ms β 65.33 ms β +1.05x faster β
β QQuery 37 β 37.09 ms β 37.36 ms β no change β
β QQuery 38 β 42.89 ms β 41.51 ms β no change β
β QQuery 39 β 144.34 ms β 152.11 ms β 1.05x slower β
β QQuery 40 β 15.34 ms β 15.15 ms β no change β
β QQuery 41 β 14.73 ms β 14.51 ms β no change β
β QQuery 42 β 14.38 ms β 14.17 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 20071.70ms β
β Total Time (spill-dedup-view-arrays) β 20288.93ms β
β Average Time (HEAD) β 466.78ms β
β Average Time (spill-dedup-view-arrays) β 471.84ms β
β Queries Faster β 2 β
β Queries Slower β 2 β
β Queries with No Change β 39 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.28 / 4.25 Β±5.80 / 15.85 ms β 1.28 / 4.29 Β±5.88 / 16.05 ms β no change β
β QQuery 1 β 12.25 / 12.37 Β±0.11 / 12.57 ms β 12.01 / 12.36 Β±0.20 / 12.60 ms β no change β
β QQuery 2 β 37.34 / 37.73 Β±0.41 / 38.43 ms β 36.83 / 37.30 Β±0.34 / 37.75 ms β no change β
β QQuery 3 β 33.04 / 33.91 Β±1.01 / 35.85 ms β 31.59 / 32.35 Β±0.64 / 33.44 ms β no change β
β QQuery 4 β 252.42 / 256.86 Β±3.78 / 263.45 ms β 250.67 / 259.36 Β±6.03 / 268.87 ms β no change β
β QQuery 5 β 281.67 / 283.98 Β±2.29 / 288.11 ms β 282.29 / 287.18 Β±4.11 / 291.90 ms β no change β
β QQuery 6 β 1.28 / 1.43 Β±0.23 / 1.88 ms β 1.34 / 1.48 Β±0.23 / 1.94 ms β no change β
β QQuery 7 β 12.81 / 13.07 Β±0.20 / 13.40 ms β 13.24 / 13.49 Β±0.23 / 13.84 ms β no change β
β QQuery 8 β 347.37 / 360.42 Β±11.73 / 376.28 ms β 353.59 / 361.49 Β±7.59 / 374.80 ms β no change β
β QQuery 9 β 489.29 / 507.98 Β±11.99 / 522.11 ms β 488.81 / 515.20 Β±16.85 / 540.84 ms β no change β
β QQuery 10 β 65.19 / 68.28 Β±4.20 / 76.31 ms β 66.13 / 68.32 Β±1.14 / 69.25 ms β no change β
β QQuery 11 β 75.86 / 77.33 Β±1.16 / 79.36 ms β 76.45 / 78.59 Β±1.57 / 80.69 ms β no change β
β QQuery 12 β 267.11 / 287.45 Β±17.41 / 319.73 ms β 277.32 / 282.44 Β±5.20 / 292.39 ms β no change β
β QQuery 13 β 385.73 / 402.04 Β±10.44 / 414.73 ms β 376.75 / 400.33 Β±18.37 / 432.91 ms β no change β
β QQuery 14 β 280.92 / 294.70 Β±12.64 / 316.32 ms β 294.05 / 301.66 Β±5.69 / 311.29 ms β no change β
β QQuery 15 β 289.54 / 303.43 Β±14.44 / 328.29 ms β 299.84 / 314.85 Β±10.89 / 328.35 ms β no change β
β QQuery 16 β 640.17 / 655.43 Β±10.50 / 668.46 ms β 644.58 / 673.11 Β±17.57 / 692.04 ms β no change β
β QQuery 17 β 659.43 / 667.35 Β±9.78 / 685.31 ms β 653.97 / 664.38 Β±9.31 / 678.37 ms β no change β
β QQuery 18 β 1359.44 / 1378.61 Β±17.83 / 1407.07 ms β 1315.20 / 1351.95 Β±25.96 / 1388.54 ms β no change β
β QQuery 19 β 28.96 / 29.63 Β±0.63 / 30.82 ms β 27.44 / 30.26 Β±5.15 / 40.55 ms β no change β
β QQuery 20 β 521.31 / 528.39 Β±7.67 / 541.61 ms β 527.01 / 530.86 Β±3.45 / 535.01 ms β no change β
β QQuery 21 β 522.04 / 524.25 Β±2.53 / 528.92 ms β 537.00 / 546.57 Β±8.00 / 557.31 ms β no change β
β QQuery 22 β 1013.84 / 1068.56 Β±57.90 / 1172.35 ms β 1024.78 / 1046.45 Β±19.01 / 1074.78 ms β no change β
β QQuery 23 β 3111.93 / 3286.39 Β±102.46 / 3418.14 ms β 3190.84 / 3425.89 Β±165.58 / 3609.41 ms β no change β
β QQuery 24 β 41.91 / 99.75 Β±75.65 / 241.06 ms β 40.44 / 74.66 Β±47.90 / 164.94 ms β +1.34x faster β
β QQuery 25 β 104.85 / 144.70 Β±46.89 / 215.65 ms β 107.94 / 161.06 Β±99.47 / 359.91 ms β 1.11x slower β
β QQuery 26 β 41.94 / 43.34 Β±1.30 / 45.37 ms β 42.11 / 42.85 Β±0.61 / 43.63 ms β no change β
β QQuery 27 β 524.08 / 533.59 Β±8.58 / 544.92 ms β 528.92 / 533.90 Β±4.36 / 539.68 ms β no change β
β QQuery 28 β 2965.01 / 2988.90 Β±23.43 / 3025.07 ms β 2969.80 / 3015.75 Β±40.47 / 3069.29 ms β no change β
β QQuery 29 β 42.48 / 46.29 Β±7.02 / 60.33 ms β 42.46 / 52.75 Β±18.02 / 88.61 ms β 1.14x slower β
β QQuery 30 β 313.42 / 322.47 Β±5.01 / 327.93 ms β 328.08 / 332.89 Β±4.69 / 341.08 ms β no change β
β QQuery 31 β 302.02 / 307.31 Β±5.32 / 315.85 ms β 290.50 / 303.87 Β±7.03 / 311.35 ms β no change β
β QQuery 32 β 1136.10 / 1220.36 Β±58.73 / 1317.49 ms β 1127.80 / 1265.00 Β±97.38 / 1407.11 ms β no change β
β QQuery 33 β 1605.93 / 1702.30 Β±53.58 / 1765.84 ms β 1695.52 / 1715.34 Β±23.17 / 1756.09 ms β no change β
β QQuery 34 β 1646.71 / 1724.91 Β±42.09 / 1772.83 ms β 1667.15 / 1811.90 Β±200.32 / 2207.25 ms β 1.05x slower β
β QQuery 35 β 319.39 / 432.85 Β±154.60 / 734.87 ms β 325.07 / 429.25 Β±172.11 / 768.88 ms β no change β
β QQuery 36 β 68.88 / 79.03 Β±12.38 / 102.35 ms β 65.33 / 75.22 Β±6.21 / 84.75 ms β no change β
β QQuery 37 β 37.09 / 39.96 Β±4.36 / 48.58 ms β 37.36 / 39.19 Β±2.59 / 44.26 ms β no change β
β QQuery 38 β 42.89 / 48.85 Β±5.41 / 57.23 ms β 41.51 / 43.70 Β±1.89 / 46.46 ms β +1.12x faster β
β QQuery 39 β 144.34 / 156.46 Β±10.66 / 176.19 ms β 152.11 / 161.63 Β±12.02 / 184.83 ms β no change β
β QQuery 40 β 15.34 / 16.06 Β±0.45 / 16.66 ms β 15.15 / 15.60 Β±0.33 / 16.01 ms β no change β
β QQuery 41 β 14.73 / 15.15 Β±0.21 / 15.30 ms β 14.51 / 15.32 Β±0.92 / 16.81 ms β no change β
β QQuery 42 β 14.38 / 16.95 Β±4.94 / 26.82 ms β 14.17 / 14.33 Β±0.15 / 14.55 ms β +1.18x faster β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 21023.08ms β
β Total Time (spill-dedup-view-arrays) β 21344.38ms β
β Average Time (HEAD) β 488.91ms β
β Average Time (spill-dedup-view-arrays) β 496.38ms β
β Queries Faster β 3 β
β Queries Slower β 3 β
β Queries with No Change β 37 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 17.3 GiB |
| Avg memory | 6.0 GiB |
| CPU user | 1037.1s |
| CPU sys | 104.6s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 18.2 GiB |
| Avg memory | 5.8 GiB |
| CPU user | 1040.2s |
| CPU sys | 108.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783807092
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.31 ms β 40.50 ms β no change β
β QQuery 2 β 19.08 ms β 18.44 ms β no change β
β QQuery 3 β 29.27 ms β 27.95 ms β no change β
β QQuery 4 β 17.85 ms β 17.17 ms β no change β
β QQuery 5 β 36.25 ms β 34.70 ms β no change β
β QQuery 6 β 16.54 ms β 16.13 ms β no change β
β QQuery 7 β 41.89 ms β 41.00 ms β no change β
β QQuery 8 β 40.93 ms β 39.75 ms β no change β
β QQuery 9 β 48.99 ms β 49.24 ms β no change β
β QQuery 10 β 40.31 ms β 40.88 ms β no change β
β QQuery 11 β 13.54 ms β 13.35 ms β no change β
β QQuery 12 β 21.16 ms β 20.60 ms β no change β
β QQuery 13 β 39.77 ms β 39.50 ms β no change β
β QQuery 14 β 24.26 ms β 24.38 ms β no change β
β QQuery 15 β 30.79 ms β 30.30 ms β no change β
β QQuery 16 β 14.08 ms β 13.81 ms β no change β
β QQuery 17 β 73.81 ms β 71.52 ms β no change β
β QQuery 18 β 59.60 ms β 60.76 ms β no change β
β QQuery 19 β 31.64 ms β 31.90 ms β no change β
β QQuery 20 β 31.74 ms β 31.37 ms β no change β
β QQuery 21 β 54.60 ms β 55.62 ms β no change β
β QQuery 22 β 13.60 ms β 13.61 ms β no change β
βββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 741.03ms β
β Total Time (spill-dedup-view-arrays) β 732.48ms β
β Average Time (HEAD) β 33.68ms β
β Average Time (spill-dedup-view-arrays) β 33.29ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 22 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.31 / 42.53 Β±2.19 / 46.91 ms β 40.50 / 41.34 Β±1.01 / 43.11 ms β no change β
β QQuery 2 β 19.08 / 19.32 Β±0.34 / 19.98 ms β 18.44 / 18.88 Β±0.50 / 19.79 ms β no change β
β QQuery 3 β 29.27 / 30.22 Β±0.99 / 31.99 ms β 27.95 / 29.18 Β±1.07 / 30.88 ms β no change β
β QQuery 4 β 17.85 / 18.27 Β±0.50 / 19.24 ms β 17.17 / 17.33 Β±0.14 / 17.50 ms β +1.05x faster β
β QQuery 5 β 36.25 / 36.39 Β±0.13 / 36.61 ms β 34.70 / 34.89 Β±0.22 / 35.32 ms β no change β
β QQuery 6 β 16.54 / 16.70 Β±0.09 / 16.80 ms β 16.13 / 16.34 Β±0.16 / 16.61 ms β no change β
β QQuery 7 β 41.89 / 44.12 Β±2.54 / 48.78 ms β 41.00 / 42.22 Β±1.09 / 44.22 ms β no change β
β QQuery 8 β 40.93 / 41.48 Β±0.61 / 42.66 ms β 39.75 / 40.57 Β±0.83 / 41.97 ms β no change β
β QQuery 9 β 48.99 / 50.81 Β±1.40 / 52.95 ms β 49.24 / 50.62 Β±1.57 / 53.48 ms β no change β
β QQuery 10 β 40.31 / 41.05 Β±0.84 / 42.62 ms β 40.88 / 41.34 Β±0.33 / 41.70 ms β no change β
β QQuery 11 β 13.54 / 13.74 Β±0.20 / 14.03 ms β 13.35 / 13.55 Β±0.11 / 13.64 ms β no change β
β QQuery 12 β 21.16 / 21.43 Β±0.21 / 21.78 ms β 20.60 / 21.15 Β±0.37 / 21.73 ms β no change β
β QQuery 13 β 39.77 / 40.95 Β±0.71 / 42.00 ms β 39.50 / 40.64 Β±1.40 / 42.93 ms β no change β
β QQuery 14 β 24.26 / 24.54 Β±0.24 / 24.99 ms β 24.38 / 24.58 Β±0.13 / 24.76 ms β no change β
β QQuery 15 β 30.79 / 31.11 Β±0.61 / 32.32 ms β 30.30 / 30.75 Β±0.38 / 31.44 ms β no change β
β QQuery 16 β 14.08 / 14.29 Β±0.18 / 14.55 ms β 13.81 / 14.06 Β±0.20 / 14.36 ms β no change β
β QQuery 17 β 73.81 / 76.45 Β±2.07 / 80.08 ms β 71.52 / 72.80 Β±0.95 / 74.06 ms β no change β
β QQuery 18 β 59.60 / 61.87 Β±1.62 / 64.31 ms β 60.76 / 62.98 Β±1.21 / 63.95 ms β no change β
β QQuery 19 β 31.64 / 32.52 Β±1.38 / 35.28 ms β 31.90 / 32.20 Β±0.33 / 32.84 ms β no change β
β QQuery 20 β 31.74 / 31.92 Β±0.18 / 32.27 ms β 31.37 / 32.06 Β±0.60 / 33.18 ms β no change β
β QQuery 21 β 54.60 / 55.65 Β±0.86 / 56.79 ms β 55.62 / 56.07 Β±0.41 / 56.74 ms β no change β
β QQuery 22 β 13.60 / 13.78 Β±0.15 / 14.03 ms β 13.61 / 13.86 Β±0.23 / 14.29 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 759.13ms β
β Total Time (spill-dedup-view-arrays) β 747.40ms β
β Average Time (HEAD) β 34.51ms β
β Average Time (spill-dedup-view-arrays) β 33.97ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 21 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 512.9 MiB |
| CPU user | 21.3s |
| CPU sys | 1.8s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 521.5 MiB |
| CPU user | 20.9s |
| CPU sys | 1.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783919431
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783730184)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783730184-2606-c2qjr 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5786386667
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5786320092)
**Instance:** `c4a-highmem-32` (16 vCPU / 128 GiB) | `Linux bench-c5786320092-2619-rlmhk 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 32
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 2 MiB (32 instances)
L1i cache: 2 MiB (32 instances)
L2 cache: 64 MiB (32 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
resources:
cpu: "16"
memory: "128Gi"
arch: "arm64"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789423405
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789126649)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.67 ms β 134.83 ms β no change β
β Q2 β 137.51 ms β 137.35 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 199.81 ms β 198.92 ms β no change β
β Q5 β 229.06 ms β 225.85 ms β no change β
β Q6 β 250.51 ms β 251.28 ms β no change β
β Q7 β 676.15 ms β 683.92 ms β no change β
β Q8 β 493.33 ms β 513.26 ms β no change β
β Q9 β 530.58 ms β 559.93 ms β 1.06x slower β
β Q10 β 1234.71 ms β 1256.88 ms β no change β
β Q11 β 375.28 ms β 415.56 ms β 1.11x slower β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4262.61ms β
β Total Time (spill-dedup-view-arrays) β 4377.78ms β
β Average Time (HEAD) β 426.26ms β
β Average Time (spill-dedup-view-arrays) β 437.78ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 8 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.67 / 136.38 Β±0.94 / 138.24 ms β 134.83 / 135.61 Β±1.04 / 137.65 ms β no change β
β Q2 β 137.51 / 139.77 Β±1.98 / 143.20 ms β 137.35 / 138.57 Β±1.00 / 140.33 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 199.81 / 201.07 Β±1.30 / 203.41 ms β 198.92 / 204.35 Β±4.41 / 211.43 ms β no change β
β Q5 β 229.06 / 232.21 Β±1.79 / 234.26 ms β 225.85 / 228.38 Β±2.72 / 232.65 ms β no change β
β Q6 β 250.51 / 253.66 Β±1.96 / 256.45 ms β 251.28 / 253.25 Β±1.22 / 254.54 ms β no change β
β Q7 β 676.15 / 687.44 Β±6.71 / 696.59 ms β 683.92 / 692.76 Β±5.00 / 698.96 ms β no change β
β Q8 β 493.33 / 497.31 Β±4.96 / 506.71 ms β 513.26 / 513.94 Β±0.49 / 514.60 ms β no change β
β Q9 β 530.58 / 538.60 Β±7.94 / 550.73 ms β 559.93 / 567.99 Β±5.39 / 575.85 ms β 1.05x slower β
β Q10 β 1234.71 / 1248.41 Β±9.41 / 1257.34 ms β 1256.88 / 1270.27 Β±10.16 / 1282.45 ms β no change β
β Q11 β 375.28 / 381.28 Β±4.67 / 386.85 ms β 415.56 / 418.33 Β±3.33 / 424.57 ms β 1.10x slower β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4316.14ms β
β Total Time (spill-dedup-view-arrays) β 4423.44ms β
β Average Time (HEAD) β 431.61ms β
β Average Time (spill-dedup-view-arrays) β 442.34ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 8 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `207d7f3 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.6 MiB | 175.3 MiB | -0.2% |
| 2 | 219.1 MiB | 219.9 MiB | +0.3% |
| 3 | 552.2 MiB | 516.4 MiB | -6.5% |
| 4 | 264.9 MiB | 266.1 MiB | +0.4% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 356.1 MiB | 355.7 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 499.8 MiB | +0.4% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 523.9 MiB | 540.1 MiB | +3.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`207d7f3 (merge-base)`) | 552.2 MiB | 1.3 GiB | 770.5 MiB | 2.4Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 540.1 MiB | 1.1 GiB | 549.9 MiB | 2.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 943.2 MiB |
| CPU user | 66.3s |
| CPU sys | 16.1s |
| Peak spill | 1012.0 MiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 807.0 MiB |
| CPU user | 66.5s |
| CPU sys | 15.8s |
| Peak spill | 1.1 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971
run benchmarks
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783372866
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.23 ms β 1.27 ms β no change β
β QQuery 1 β 11.66 ms β 12.12 ms β no change β
β QQuery 2 β 36.73 ms β 37.18 ms β no change β
β QQuery 3 β 31.65 ms β 31.98 ms β no change β
β QQuery 4 β 254.70 ms β 257.66 ms β no change β
β QQuery 5 β 281.07 ms β 286.27 ms β no change β
β QQuery 6 β 1.35 ms β 1.39 ms β no change β
β QQuery 7 β 13.21 ms β 13.55 ms β no change β
β QQuery 8 β 352.03 ms β 355.90 ms β no change β
β QQuery 9 β 470.51 ms β 505.45 ms β 1.07x slower β
β QQuery 10 β 64.48 ms β 67.20 ms β no change β
β QQuery 11 β 75.86 ms β 78.03 ms β no change β
β QQuery 12 β 263.82 ms β 281.24 ms β 1.07x slower β
β QQuery 13 β 360.87 ms β 373.11 ms β no change β
β QQuery 14 β 277.35 ms β 289.75 ms β no change β
β QQuery 15 β 281.43 ms β 311.77 ms β 1.11x slower β
β QQuery 16 β 647.93 ms β 653.53 ms β no change β
β QQuery 17 β 661.13 ms β 632.19 ms β no change β
β QQuery 18 β 1335.60 ms β 1302.79 ms β no change β
β QQuery 19 β 28.54 ms β 27.80 ms β no change β
β QQuery 20 β 522.53 ms β 517.25 ms β no change β
β QQuery 21 β 520.87 ms β 524.00 ms β no change β
β QQuery 22 β 998.97 ms β 992.71 ms β no change β
β QQuery 23 β 3121.63 ms β 3201.18 ms β no change β
β QQuery 24 β 41.52 ms β 40.97 ms β no change β
β QQuery 25 β 107.03 ms β 109.19 ms β no change β
β QQuery 26 β 41.67 ms β 42.23 ms β no change β
β QQuery 27 β 518.07 ms β 526.94 ms β no change β
β QQuery 28 β 2858.57 ms β 2925.60 ms β no change β
β QQuery 29 β 41.85 ms β 42.01 ms β no change β
β QQuery 30 β 304.69 ms β 325.14 ms β 1.07x slower β
β QQuery 31 β 304.43 ms β 301.93 ms β no change β
β QQuery 32 β 1153.63 ms β 1264.73 ms β 1.10x slower β
β QQuery 33 β 1704.97 ms β 1691.74 ms β no change β
β QQuery 34 β 1620.69 ms β 1668.76 ms β no change β
β QQuery 35 β 303.29 ms β 331.58 ms β 1.09x slower β
β QQuery 36 β 68.60 ms β 76.14 ms β 1.11x slower β
β QQuery 37 β 35.78 ms β 37.37 ms β no change β
β QQuery 38 β 41.91 ms β 46.54 ms β 1.11x slower β
β QQuery 39 β 146.81 ms β 151.02 ms β no change β
β QQuery 40 β 15.04 ms β 15.15 ms β no change β
β QQuery 41 β 14.38 ms β 14.44 ms β no change β
β QQuery 42 β 13.77 ms β 13.87 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 19951.87ms β
β Total Time (spill-dedup-view-arrays) β 20380.72ms β
β Average Time (HEAD) β 464.00ms β
β Average Time (spill-dedup-view-arrays) β 473.97ms β
β Queries Faster β 0 β
β Queries Slower β 8 β
β Queries with No Change β 35 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.23 / 4.10 Β±5.58 / 15.27 ms β 1.27 / 4.19 Β±5.69 / 15.57 ms β no change β
β QQuery 1 β 11.66 / 11.96 Β±0.17 / 12.12 ms β 12.12 / 12.29 Β±0.10 / 12.43 ms β no change β
β QQuery 2 β 36.73 / 37.13 Β±0.36 / 37.66 ms β 37.18 / 37.57 Β±0.32 / 38.07 ms β no change β
β QQuery 3 β 31.65 / 32.18 Β±0.62 / 33.37 ms β 31.98 / 32.36 Β±0.35 / 32.94 ms β no change β
β QQuery 4 β 254.70 / 257.53 Β±3.94 / 265.28 ms β 257.66 / 261.71 Β±3.58 / 267.24 ms β no change β
β QQuery 5 β 281.07 / 284.54 Β±3.75 / 290.27 ms β 286.27 / 292.81 Β±4.46 / 299.72 ms β no change β
β QQuery 6 β 1.35 / 1.50 Β±0.22 / 1.93 ms β 1.39 / 1.54 Β±0.22 / 1.96 ms β no change β
β QQuery 7 β 13.21 / 13.56 Β±0.22 / 13.91 ms β 13.55 / 13.64 Β±0.06 / 13.71 ms β no change β
β QQuery 8 β 352.03 / 357.54 Β±4.76 / 365.71 ms β 355.90 / 363.61 Β±7.75 / 377.43 ms β no change β
β QQuery 9 β 470.51 / 485.86 Β±9.95 / 496.16 ms β 505.45 / 519.56 Β±21.58 / 562.30 ms β 1.07x slower β
β QQuery 10 β 64.48 / 65.35 Β±0.77 / 66.56 ms β 67.20 / 68.34 Β±0.99 / 69.91 ms β no change β
β QQuery 11 β 75.86 / 76.57 Β±0.63 / 77.63 ms β 78.03 / 78.33 Β±0.18 / 78.56 ms β no change β
β QQuery 12 β 263.82 / 270.93 Β±5.75 / 278.88 ms β 281.24 / 288.75 Β±5.27 / 297.56 ms β 1.07x slower β
β QQuery 13 β 360.87 / 386.14 Β±16.08 / 407.17 ms β 373.11 / 396.35 Β±17.48 / 412.88 ms β no change β
β QQuery 14 β 277.35 / 284.04 Β±4.83 / 290.51 ms β 289.75 / 312.82 Β±28.46 / 368.87 ms β 1.10x slower β
β QQuery 15 β 281.43 / 309.98 Β±15.52 / 328.76 ms β 311.77 / 319.67 Β±5.74 / 325.11 ms β no change β
β QQuery 16 β 647.93 / 658.30 Β±7.93 / 671.87 ms β 653.53 / 663.15 Β±6.64 / 672.67 ms β no change β
β QQuery 17 β 661.13 / 665.78 Β±4.29 / 673.81 ms β 632.19 / 654.86 Β±13.62 / 670.11 ms β no change β
β QQuery 18 β 1335.60 / 1352.99 Β±14.75 / 1377.25 ms β 1302.79 / 1351.70 Β±35.91 / 1401.10 ms β no change β
β QQuery 19 β 28.54 / 32.06 Β±5.16 / 42.29 ms β 27.80 / 29.97 Β±3.63 / 37.17 ms β +1.07x faster β
β QQuery 20 β 522.53 / 531.01 Β±10.48 / 548.24 ms β 517.25 / 534.95 Β±11.39 / 550.64 ms β no change β
β QQuery 21 β 520.87 / 533.87 Β±13.80 / 560.53 ms β 524.00 / 529.70 Β±6.33 / 540.10 ms β no change β
β QQuery 22 β 998.97 / 1013.28 Β±13.96 / 1031.79 ms β 992.71 / 1030.29 Β±21.83 / 1059.47 ms β no change β
β QQuery 23 β 3121.63 / 3456.03 Β±226.59 / 3796.78 ms β 3201.18 / 3420.20 Β±172.14 / 3677.26 ms β no change β
β QQuery 24 β 41.52 / 47.92 Β±10.80 / 69.49 ms β 40.97 / 84.43 Β±64.51 / 209.23 ms β 1.76x slower β
β QQuery 25 β 107.03 / 202.16 Β±185.21 / 572.56 ms β 109.19 / 140.52 Β±52.65 / 244.99 ms β +1.44x faster β
β QQuery 26 β 41.67 / 42.32 Β±0.41 / 42.81 ms β 42.23 / 44.16 Β±2.32 / 48.52 ms β no change β
β QQuery 27 β 518.07 / 531.85 Β±7.87 / 542.66 ms β 526.94 / 533.93 Β±5.53 / 541.64 ms β no change β
β QQuery 28 β 2858.57 / 2908.83 Β±45.27 / 2978.22 ms β 2925.60 / 2960.02 Β±21.44 / 2990.06 ms β no change β
β QQuery 29 β 41.85 / 45.82 Β±6.99 / 59.78 ms β 42.01 / 44.07 Β±2.70 / 49.40 ms β no change β
β QQuery 30 β 304.69 / 325.08 Β±13.73 / 343.31 ms β 325.14 / 346.03 Β±19.42 / 379.89 ms β 1.06x slower β
β QQuery 31 β 304.43 / 314.04 Β±10.72 / 333.49 ms β 301.93 / 316.76 Β±13.52 / 339.08 ms β no change β
β QQuery 32 β 1153.63 / 1212.70 Β±66.24 / 1339.90 ms β 1264.73 / 1390.24 Β±141.14 / 1635.00 ms β 1.15x slower β
β QQuery 33 β 1704.97 / 1734.56 Β±18.02 / 1755.33 ms β 1691.74 / 1774.50 Β±132.87 / 2039.55 ms β no change β
β QQuery 34 β 1620.69 / 1749.17 Β±147.09 / 2028.49 ms β 1668.76 / 1776.49 Β±96.54 / 1916.51 ms β no change β
β QQuery 35 β 303.29 / 378.03 Β±123.34 / 623.48 ms β 331.58 / 415.24 Β±129.89 / 673.02 ms β 1.10x slower β
β QQuery 36 β 68.60 / 74.95 Β±6.38 / 85.99 ms β 76.14 / 78.72 Β±2.02 / 82.26 ms β 1.05x slower β
β QQuery 37 β 35.78 / 39.89 Β±6.20 / 52.16 ms β 37.37 / 44.04 Β±8.32 / 58.56 ms β 1.10x slower β
β QQuery 38 β 41.91 / 43.12 Β±1.22 / 44.81 ms β 46.54 / 53.93 Β±5.00 / 60.12 ms β 1.25x slower β
β QQuery 39 β 146.81 / 156.39 Β±7.97 / 170.69 ms β 151.02 / 162.93 Β±11.47 / 183.28 ms β no change β
β QQuery 40 β 15.04 / 15.37 Β±0.27 / 15.64 ms β 15.15 / 17.39 Β±2.87 / 22.72 ms β 1.13x slower β
β QQuery 41 β 14.38 / 15.69 Β±1.92 / 19.48 ms β 14.44 / 16.58 Β±3.63 / 23.82 ms β 1.06x slower β
β QQuery 42 β 13.77 / 13.91 Β±0.08 / 14.02 ms β 13.87 / 15.53 Β±3.01 / 21.54 ms β 1.12x slower β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 20974.03ms β
β Total Time (spill-dedup-view-arrays) β 21433.88ms β
β Average Time (HEAD) β 487.77ms β
β Average Time (spill-dedup-view-arrays) β 498.46ms β
β Queries Faster β 2 β
β Queries Slower β 13 β
β Queries with No Change β 28 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 17.5 GiB |
| Avg memory | 6.1 GiB |
| CPU user | 1029.2s |
| CPU sys | 103.3s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 18.4 GiB |
| Avg memory | 6.1 GiB |
| CPU user | 1042.6s |
| CPU sys | 109.4s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863265538
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863112345)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863112345-2865-h9vwd 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867396562
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870814838
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 ms β 1.22 ms β no change β
β QQuery 1 β 11.78 ms β 11.70 ms β no change β
β QQuery 2 β 37.10 ms β 37.23 ms β no change β
β QQuery 3 β 31.31 ms β 31.36 ms β no change β
β QQuery 4 β 329.84 ms β 332.95 ms β no change β
β QQuery 5 β 515.26 ms β 530.39 ms β no change β
β QQuery 6 β 1.28 ms β 1.28 ms β no change β
β QQuery 7 β 12.82 ms β 12.80 ms β no change β
β QQuery 8 β 691.14 ms β 698.62 ms β no change β
β QQuery 9 β 573.73 ms β 591.03 ms β no change β
β QQuery 10 β 65.37 ms β 65.49 ms β no change β
β QQuery 11 β 75.82 ms β 76.38 ms β no change β
β QQuery 12 β 558.18 ms β 564.17 ms β no change β
β QQuery 13 β 1501.29 ms β 1533.93 ms β no change β
β QQuery 14 β 621.10 ms β 621.18 ms β no change β
β QQuery 15 β 374.69 ms β 370.71 ms β no change β
β QQuery 16 β 1306.28 ms β 1315.75 ms β no change β
β QQuery 17 β 942.31 ms β 930.31 ms β no change β
β QQuery 18 β 2731.40 ms β 2719.93 ms β no change β
β QQuery 19 β 27.72 ms β 27.31 ms β no change β
β QQuery 20 β 512.11 ms β 509.42 ms β no change β
β QQuery 21 β 499.84 ms β 497.95 ms β no change β
β QQuery 22 β 968.31 ms β 969.45 ms β no change β
β QQuery 23 β FAIL β FAIL β incomparable β
β QQuery 24 β 40.41 ms β 40.19 ms β no change β
β QQuery 25 β 105.26 ms β 104.28 ms β no change β
β QQuery 26 β 40.82 ms β 42.63 ms β no change β
β QQuery 27 β 514.42 ms β 507.28 ms β no change β
β QQuery 28 β 3373.94 ms β 3323.85 ms β no change β
β QQuery 29 β 41.95 ms β 41.92 ms β no change β
β QQuery 30 β 438.82 ms β 434.37 ms β no change β
β QQuery 31 β 531.96 ms β 524.00 ms β no change β
β QQuery 32 β 3494.58 ms β 3505.65 ms β no change β
β QQuery 33 β 4294.38 ms β 4346.00 ms β no change β
β QQuery 34 β 4290.76 ms β 4394.65 ms β no change β
β QQuery 35 β 337.05 ms β 333.52 ms β no change β
β QQuery 36 β 64.25 ms β 65.55 ms β no change β
β QQuery 37 β 35.55 ms β 35.51 ms β no change β
β QQuery 38 β 40.81 ms β 42.08 ms β no change β
β QQuery 39 β 136.44 ms β 131.19 ms β no change β
β QQuery 40 β 14.18 ms β 13.88 ms β no change β
β QQuery 41 β 13.55 ms β 13.42 ms β no change β
β QQuery 42 β 12.98 ms β 12.92 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 30212.03ms β
β Total Time (spill-dedup-view-arrays) β 30363.42ms β
β Average Time (HEAD) β 719.33ms β
β Average Time (spill-dedup-view-arrays) β 722.94ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 42 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 / 4.04 Β±5.50 / 15.04 ms β 1.22 / 3.92 Β±5.33 / 14.58 ms β no change β
β QQuery 1 β 11.78 / 11.83 Β±0.06 / 11.96 ms β 11.70 / 11.85 Β±0.08 / 11.94 ms β no change β
β QQuery 2 β 37.10 / 37.55 Β±0.27 / 37.80 ms β 37.23 / 37.48 Β±0.15 / 37.64 ms β no change β
β QQuery 3 β 31.31 / 32.23 Β±0.82 / 33.58 ms β 31.36 / 31.66 Β±0.24 / 31.94 ms β no change β
β QQuery 4 β 329.84 / 335.53 Β±3.79 / 340.27 ms β 332.95 / 340.61 Β±5.82 / 348.12 ms β no change β
β QQuery 5 β 515.26 / 526.19 Β±7.12 / 534.58 ms β 530.39 / 535.19 Β±3.35 / 540.90 ms β no change β
β QQuery 6 β 1.28 / 1.44 Β±0.24 / 1.91 ms β 1.28 / 1.43 Β±0.23 / 1.89 ms β no change β
β QQuery 7 β 12.82 / 13.17 Β±0.18 / 13.33 ms β 12.80 / 12.87 Β±0.07 / 12.97 ms β no change β
β QQuery 8 β 691.14 / 702.73 Β±10.11 / 715.35 ms β 698.62 / 707.55 Β±8.04 / 722.44 ms β no change β
β QQuery 9 β 573.73 / 585.79 Β±11.82 / 605.36 ms β 591.03 / 604.89 Β±10.45 / 618.20 ms β no change β
β QQuery 10 β 65.37 / 67.13 Β±1.50 / 69.11 ms β 65.49 / 67.78 Β±1.29 / 69.38 ms β no change β
β QQuery 11 β 75.82 / 78.57 Β±1.79 / 80.86 ms β 76.38 / 78.76 Β±1.77 / 80.53 ms β no change β
β QQuery 12 β 558.18 / 572.51 Β±15.61 / 596.56 ms β 564.17 / 576.36 Β±12.07 / 596.45 ms β no change β
β QQuery 13 β 1501.29 / 1533.91 Β±27.79 / 1569.67 ms β 1533.93 / 1550.20 Β±21.76 / 1591.71 ms β no change β
β QQuery 14 β 621.10 / 637.22 Β±14.28 / 660.83 ms β 621.18 / 649.29 Β±14.86 / 663.91 ms β no change β
β QQuery 15 β 374.69 / 378.96 Β±3.47 / 384.80 ms β 370.71 / 381.76 Β±7.06 / 390.91 ms β no change β
β QQuery 16 β 1306.28 / 1321.61 Β±11.28 / 1335.19 ms β 1315.75 / 1331.16 Β±8.18 / 1338.42 ms β no change β
β QQuery 17 β 942.31 / 961.46 Β±11.09 / 973.99 ms β 930.31 / 962.18 Β±19.58 / 985.88 ms β no change β
β QQuery 18 β 2731.40 / 2762.65 Β±28.15 / 2799.90 ms β 2719.93 / 2767.96 Β±26.20 / 2793.26 ms β no change β
β QQuery 19 β 27.72 / 28.14 Β±0.61 / 29.34 ms β 27.31 / 27.74 Β±0.34 / 28.28 ms β no change β
β QQuery 20 β 512.11 / 518.26 Β±6.79 / 531.47 ms β 509.42 / 512.19 Β±2.13 / 514.17 ms β no change β
β QQuery 21 β 499.84 / 503.09 Β±3.41 / 509.08 ms β 497.95 / 510.94 Β±9.42 / 521.84 ms β no change β
β QQuery 22 β 968.31 / 973.54 Β±3.04 / 977.49 ms β 969.45 / 974.52 Β±3.85 / 979.46 ms β no change β
β QQuery 23 β FAIL β FAIL β incomparable β
β QQuery 24 β 40.41 / 50.75 Β±9.77 / 66.04 ms β 40.19 / 44.80 Β±7.03 / 58.60 ms β +1.13x faster β
β QQuery 25 β 105.26 / 120.59 Β±20.04 / 160.04 ms β 104.28 / 108.92 Β±3.44 / 113.91 ms β +1.11x faster β
β QQuery 26 β 40.82 / 41.03 Β±0.16 / 41.20 ms β 42.63 / 54.03 Β±17.11 / 87.95 ms β 1.32x slower β
β QQuery 27 β 514.42 / 521.56 Β±4.63 / 528.11 ms β 507.28 / 514.03 Β±4.25 / 520.46 ms β no change β
β QQuery 28 β 3373.94 / 3386.43 Β±7.94 / 3395.49 ms β 3323.85 / 3362.82 Β±25.63 / 3400.67 ms β no change β
β QQuery 29 β 41.95 / 42.31 Β±0.35 / 42.95 ms β 41.92 / 43.28 Β±2.45 / 48.18 ms β no change β
β QQuery 30 β 438.82 / 446.74 Β±8.46 / 459.60 ms β 434.37 / 451.14 Β±13.79 / 468.17 ms β no change β
β QQuery 31 β 531.96 / 543.97 Β±9.41 / 554.91 ms β 524.00 / 537.70 Β±13.16 / 558.14 ms β no change β
β QQuery 32 β 3494.58 / 3504.73 Β±7.80 / 3513.28 ms β 3505.65 / 3514.44 Β±6.03 / 3523.08 ms β no change β
β QQuery 33 β 4294.38 / 4487.46 Β±153.94 / 4717.91 ms β 4346.00 / 4401.22 Β±46.65 / 4473.91 ms β no change β
β QQuery 34 β 4290.76 / 4410.62 Β±82.46 / 4510.00 ms β 4394.65 / 4445.95 Β±47.29 / 4529.58 ms β no change β
β QQuery 35 β 337.05 / 346.48 Β±7.90 / 358.15 ms β 333.52 / 343.70 Β±6.97 / 351.32 ms β no change β
β QQuery 36 β 64.25 / 68.77 Β±4.45 / 76.11 ms β 65.55 / 68.85 Β±5.69 / 80.20 ms β no change β
β QQuery 37 β 35.55 / 36.45 Β±0.57 / 37.35 ms β 35.51 / 36.39 Β±1.00 / 38.30 ms β no change β
β QQuery 38 β 40.81 / 44.58 Β±4.37 / 53.02 ms β 42.08 / 44.45 Β±1.94 / 46.26 ms β no change β
β QQuery 39 β 136.44 / 138.19 Β±1.84 / 141.69 ms β 131.19 / 140.04 Β±9.07 / 156.08 ms β no change β
β QQuery 40 β 14.18 / 15.10 Β±0.96 / 16.35 ms β 13.88 / 14.35 Β±0.34 / 14.87 ms β no change β
β QQuery 41 β 13.55 / 13.84 Β±0.35 / 14.48 ms β 13.42 / 14.45 Β±1.13 / 15.84 ms β no change β
β QQuery 42 β 12.98 / 13.11 Β±0.15 / 13.39 ms β 12.92 / 14.18 Β±2.05 / 18.28 ms β 1.08x slower β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 30820.29ms β
β Total Time (spill-dedup-view-arrays) β 30833.03ms β
β Average Time (HEAD) β 733.82ms β
β Average Time (spill-dedup-view-arrays) β 734.12ms β
β Queries Faster β 2 β
β Queries Slower β 2 β
β Queries with No Change β 38 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 274.3 MiB | 272.3 MiB | -0.7% |
| Query 5 | 342.7 MiB | 353.7 MiB | +3.2% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 50.2 MiB | 40.3 MiB | -19.7% |
| Query 8 | 129.8 MiB | 130.9 MiB | +0.8% |
| Query 9 | 333.6 MiB | 333.7 MiB | +0.0% |
| Query 10 | 108.7 MiB | 104.8 MiB | -3.7% |
| Query 11 | 121.5 MiB | 115.3 MiB | -5.1% |
| Query 12 | 325.9 MiB | 331.0 MiB | +1.6% |
| Query 13 | 303.3 MiB | 304.6 MiB | +0.4% |
| Query 14 | 389.5 MiB | 397.4 MiB | +2.0% |
| Query 15 | 281.6 MiB | 280.2 MiB | -0.5% |
| Query 16 | 429.8 MiB | 382.3 MiB | -11.1% |
| Query 17 | 414.8 MiB | 407.3 MiB | -1.8% |
| Query 18 | 379.0 MiB | 383.8 MiB | +1.3% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.7 MiB | +9.1% |
| Query 22 | 3.1 MiB | 3.6 MiB | +16.8% |
| Query 23 | 0 B | 0 B | 0.0% |
| Query 24 | 59.4 MiB | 60.7 MiB | +2.1% |
| Query 25 | 157.6 MiB | 174.1 MiB | +10.5% |
| Query 26 | 57.9 MiB | 57.9 MiB | +0.0% |
| Query 27 | 2.4 MiB | 2.4 MiB | +0.0% |
| Query 28 | 315.8 MiB | 314.7 MiB | -0.4% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 378.9 MiB | 392.2 MiB | +3.5% |
| Query 31 | 146.7 MiB | 148.2 MiB | +1.0% |
| Query 32 | 278.0 MiB | 288.8 MiB | +3.9% |
| Query 33 | 383.6 MiB | 359.6 MiB | -6.2% |
| Query 34 | 372.9 MiB | 370.4 MiB | -0.7% |
| Query 35 | 292.0 MiB | 294.2 MiB | +0.7% |
| Query 36 | 115.2 MiB | 107.7 MiB | -6.5% |
| Query 37 | 6.9 MiB | 6.9 MiB | +0.0% |
| Query 38 | 5.7 MiB | 5.6 MiB | -0.6% |
| Query 39 | 298.0 MiB | 283.4 MiB | -4.9% |
| Query 40 | 2.0 MiB | 2.0 MiB | -1.2% |
| Query 41 | 3.1 MiB | 3.1 MiB | +0.0% |
| Query 42 | 1.7 MiB | 1.6 MiB | -6.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`ed43e69 (merge-base)`) | 429.8 MiB | 24.2 GiB | 23.8 GiB | 57.8Γ |
| clickbench_partitioned | changed (`spill-dedup-view-arrays`) | 407.3 MiB | 22.1 GiB | 21.7 GiB | 55.5Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 24.2 GiB |
| Avg memory | 7.3 GiB |
| CPU user | 1463.4s |
| CPU sys | 307.3s |
| Peak spill | 5.2 GiB |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 22.1 GiB |
| Avg memory | 7.9 GiB |
| CPU user | 1457.9s |
| CPU sys | 309.3s |
| Peak spill | 5.2 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870639383
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5870585310-2877-4nbmw 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867397520
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863134636
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110116)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863110116-2859-z9sws 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863112345
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867799439
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867397520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.47 105.4Β±0.84ms ? ?/sec 1.00 71.5Β±0.21ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.41 100.5Β±0.80ms ? ?/sec 1.00 71.5Β±0.54ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.39 100.2Β±2.10ms ? ?/sec 1.00 71.8Β±0.45ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.01 126.2Β±1.18ms ? ?/sec 1.00 125.4Β±1.02ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.04 135.6Β±2.74ms ? ?/sec 1.00 130.9Β±3.38ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69 (merge-base)`) | 102.6 MiB | 788.7 MiB | 686.1 MiB | 7.7Γ |
| spill_views | changed (`spill-dedup-view-arrays`) | 102.6 MiB | 750.4 MiB | 647.8 MiB | 7.3Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 365.2s |
| Peak memory | 788.7 MiB |
| Avg memory | 58.9 MiB |
| CPU user | 62.6s |
| CPU sys | 29.6s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 565.2s |
| Peak memory | 750.4 MiB |
| Avg memory | 41.5 MiB |
| CPU user | 73.6s |
| CPU sys | 22.6s |
| Peak spill | 72.5 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863111696
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
baseline:
ref: ed43e69fdca8737c9d065b53b5fc5e4183302775
changed:
ref: ed43e69fdca8737c9d065b53b5fc5e4183302775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863233335
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110116)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.18 ms β 135.04 ms β no change β
β Q2 β 135.22 ms β 136.03 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.42 ms β 199.44 ms β no change β
β Q5 β 229.25 ms β 228.25 ms β no change β
β Q6 β 251.47 ms β 252.45 ms β no change β
β Q7 β 690.19 ms β 695.92 ms β no change β
β Q8 β 486.14 ms β 489.32 ms β no change β
β Q9 β 532.55 ms β 527.80 ms β no change β
β Q10 β 1235.34 ms β 1224.87 ms β no change β
β Q11 β 376.01 ms β 375.27 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4269.77ms β
β Total Time (spill-dedup-view-arrays) β 4264.39ms β
β Average Time (HEAD) β 426.98ms β
β Average Time (spill-dedup-view-arrays) β 426.44ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.18 / 136.75 Β±1.54 / 139.07 ms β 135.04 / 136.18 Β±0.86 / 137.69 ms β no change β
β Q2 β 135.22 / 135.99 Β±0.42 / 136.43 ms β 136.03 / 138.15 Β±2.11 / 140.78 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.42 / 203.72 Β±5.04 / 210.94 ms β 199.44 / 204.99 Β±5.11 / 213.05 ms β no change β
β Q5 β 229.25 / 232.17 Β±1.66 / 234.35 ms β 228.25 / 230.88 Β±1.90 / 234.16 ms β no change β
β Q6 β 251.47 / 256.05 Β±2.91 / 259.24 ms β 252.45 / 254.69 Β±1.67 / 257.58 ms β no change β
β Q7 β 690.19 / 703.02 Β±10.97 / 720.39 ms β 695.92 / 702.31 Β±5.35 / 711.12 ms β no change β
β Q8 β 486.14 / 494.06 Β±4.78 / 499.48 ms β 489.32 / 499.97 Β±7.04 / 509.39 ms β no change β
β Q9 β 532.55 / 537.68 Β±3.61 / 543.74 ms β 527.80 / 538.49 Β±8.00 / 552.43 ms β no change β
β Q10 β 1235.34 / 1254.37 Β±10.63 / 1267.25 ms β 1224.87 / 1259.94 Β±20.63 / 1287.40 ms β no change β
β Q11 β 376.01 / 379.24 Β±4.10 / 387.29 ms β 375.27 / 383.39 Β±6.45 / 394.13 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4333.06ms β
β Total Time (spill-dedup-view-arrays) β 4348.99ms β
β Average Time (HEAD) β 433.31ms β
β Average Time (spill-dedup-view-arrays) β 434.90ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.6 MiB | 175.3 MiB | -0.2% |
| 2 | 219.9 MiB | 219.9 MiB | -0.0% |
| 3 | 514.0 MiB | 511.3 MiB | -0.5% |
| 4 | 266.1 MiB | 264.9 MiB | -0.4% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.6 MiB | 356.5 MiB | +0.2% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 498.9 MiB | +0.2% |
| 10 | 500.7 MiB | 510.1 MiB | +1.9% |
| 11 | 524.2 MiB | 522.5 MiB | -0.3% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`ed43e69 (merge-base)`) | 524.2 MiB | 1.2 GiB | 670.2 MiB | 2.3Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 522.5 MiB | 1.0 GiB | 533.4 MiB | 2.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 770.7 MiB |
| CPU user | 65.3s |
| CPU sys | 16.7s |
| Peak spill | 1.2 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.0 GiB |
| Avg memory | 781.7 MiB |
| CPU user | 65.8s |
| CPU sys | 16.3s |
| Peak spill | 1.2 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867550409
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867396562)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 136.56 ms β 133.79 ms β no change β
β Q2 β 137.59 ms β 136.45 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 201.39 ms β 197.86 ms β no change β
β Q5 β 230.92 ms β 226.47 ms β no change β
β Q6 β 253.33 ms β 251.33 ms β no change β
β Q7 β 687.93 ms β 658.99 ms β no change β
β Q8 β 483.50 ms β 479.94 ms β no change β
β Q9 β 521.77 ms β 528.19 ms β no change β
β Q10 β 1209.55 ms β 1215.69 ms β no change β
β Q11 β 365.47 ms β 367.81 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4228.00ms β
β Total Time (spill-dedup-view-arrays) β 4196.51ms β
β Average Time (HEAD) β 422.80ms β
β Average Time (spill-dedup-view-arrays) β 419.65ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 136.56 / 138.54 Β±1.99 / 142.21 ms β 133.79 / 134.76 Β±1.00 / 136.66 ms β no change β
β Q2 β 137.59 / 138.87 Β±1.31 / 141.37 ms β 136.45 / 138.99 Β±1.96 / 141.41 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 201.39 / 205.84 Β±3.15 / 209.32 ms β 197.86 / 199.57 Β±1.20 / 201.57 ms β no change β
β Q5 β 230.92 / 233.35 Β±1.63 / 235.57 ms β 226.47 / 229.98 Β±2.04 / 232.23 ms β no change β
β Q6 β 253.33 / 257.42 Β±2.24 / 259.50 ms β 251.33 / 255.29 Β±3.90 / 261.63 ms β no change β
β Q7 β 687.93 / 703.31 Β±10.09 / 718.83 ms β 658.99 / 665.73 Β±8.37 / 681.86 ms β +1.06x faster β
β Q8 β 483.50 / 508.69 Β±17.56 / 532.85 ms β 479.94 / 488.06 Β±5.86 / 496.35 ms β no change β
β Q9 β 521.77 / 535.65 Β±9.39 / 543.85 ms β 528.19 / 540.41 Β±9.54 / 556.68 ms β no change β
β Q10 β 1209.55 / 1223.07 Β±8.42 / 1231.72 ms β 1215.69 / 1230.15 Β±17.24 / 1262.27 ms β no change β
β Q11 β 365.47 / 379.06 Β±10.53 / 395.51 ms β 367.81 / 370.66 Β±2.51 / 374.99 ms β no change β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4323.80ms β
β Total Time (spill-dedup-view-arrays) β 4253.61ms β
β Average Time (HEAD) β 432.38ms β
β Average Time (spill-dedup-view-arrays) β 425.36ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.1 MiB | 175.5 MiB | +0.2% |
| 2 | 219.4 MiB | 220.4 MiB | +0.4% |
| 3 | 510.9 MiB | 539.3 MiB | +5.6% |
| 4 | 266.1 MiB | 262.4 MiB | -1.4% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.7 MiB | 355.2 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 505.3 MiB | +1.5% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 523.1 MiB | 523.6 MiB | +0.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`ed43e69 (merge-base)`) | 523.1 MiB | 1.1 GiB | 581.4 MiB | 2.1Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 539.3 MiB | 1.1 GiB | 606.2 MiB | 2.1Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 797.1 MiB |
| CPU user | 66.1s |
| CPU sys | 16.1s |
| Peak spill | 1.1 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 772.1 MiB |
| CPU user | 64.0s |
| CPU sys | 15.1s |
| Peak spill | 1.2 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r4118969964
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -800,29 +804,133 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Compacts the data buffers of a view array, or returns `None` when the
+/// array is too small for compaction to be worth it.
+///
+/// Arrow's `gc()` copies the bytes of every view separately, so repeated
+/// values each get their own copy. For a dictionary-encoded Parquet column
+/// this inflates the spilled data by the average repeat count: 1M rows of
+/// 1000 distinct 64 byte values spill as 64 MB
+/// (<https://github.com/apache/datafusion/issues/23564>).
+/// [`gc_dedup_view_array`] copies each distinct value once instead, and falls
+/// back to `gc()` when the values turn out to be distinct.
+fn gc_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ if !should_gc_view_array(array) {
+ return None;
+ }
+ Some(gc_dedup_view_array(array).unwrap_or_else(|| array.gc()))
+}
+
+/// Number of non-inline values [`gc_dedup_view_array`] deduplicates before
+/// it checks whether the array has enough repeats to be worth it.
+const DEDUP_SAMPLE_VALUES: usize = 256;
+
+/// Like `gc()`, but copies each distinct value once, and zeroes null views.
+///
+/// Returns `None` if the first [`DEDUP_SAMPLE_VALUES`] non-inline values
+/// hold almost no repeats, since hashing every value then costs more than
+/// deduplication saves.
+fn gc_dedup_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ let buffers = array.data_buffers();
+ let bytes_of = |view: &ByteView| {
+ let start = view.offset as usize;
+ &buffers[view.buffer_index as usize][start..start + view.length as usize]
+ };
+ let hasher = DefaultHashBuilder::default();
+
+ // (input view, output view) of the first occurrence of each distinct value
+ let mut distinct: HashTable<(u128, u128)> = HashTable::new();
+ let mut completed: Vec<Buffer> = vec![];
+ let mut data: Vec<u8> = vec![];
+ let mut non_inline = 0;
+ let mut views = Vec::with_capacity(array.len());
+
+ for (i, &raw) in array.views().iter().enumerate() {
+ if array.is_null(i) {
+ views.push(0);
+ continue;
+ }
+ if (raw as u32) <= MAX_INLINE_VIEW_LEN {
+ views.push(raw);
+ continue;
+ }
+
+ let view = ByteView::from(raw);
+ let bytes = bytes_of(&view);
+ let hash = hasher.hash_one(bytes);
+ let found = distinct.find(hash, |(first, _)| {
+ *first == raw || bytes_of(&ByteView::from(*first)) == bytes
+ });
+ let new_view = match found {
+ Some((_, new_view)) => *new_view,
+ None => {
+ // A view offset is a `u32`, and a buffer may not exceed `i32::MAX`
+ if data.len() + bytes.len() > i32::MAX as usize {
+ completed.push(Buffer::from_vec(std::mem::take(&mut data)));
+ }
+ let new_view = ByteView {
+ buffer_index: completed.len() as u32,
+ offset: data.len() as u32,
+ ..view
+ }
+ .as_u128();
+ data.extend_from_slice(bytes);
+ distinct.insert_unique(hash, (raw, new_view), |(first, _)| {
+ hasher.hash_one(bytes_of(&ByteView::from(*first)))
+ });
+ new_view
+ }
+ };
+ views.push(new_view);
+
+ non_inline += 1;
+ if non_inline == DEDUP_SAMPLE_VALUES
Review Comment:
When labels cycle through 1,000 values, the first 256 are distinct, so we fall back to gc() despite many repeats later. Could we sample across the batch and add a test for this ordering?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789122907
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789121802
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789371962
Benchmark for [this request](https://github.com/apache/datafusion/pull/23565#issuecomment-5789125588) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `spill_views`
<details><summary>Runner log (last 40 lines)</summary>
```
Compiling datafusion-datasource-json v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-json)
Compiling datafusion-datasource-arrow v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-arrow)
Compiling datafusion-datasource-csv v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-csv)
Compiling datafusion-functions-table v55.1.0 (/workspace/datafusion-branch/datafusion/functions-table)
Compiling datafusion-catalog-listing v55.1.0 (/workspace/datafusion-branch/datafusion/catalog-listing)
Compiling datafusion-datasource-parquet v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-parquet)
Compiling datafusion-physical-optimizer v55.1.0 (/workspace/datafusion-branch/datafusion/physical-optimizer)
Compiling datafusion v55.1.0 (/workspace/datafusion-branch/datafusion/core)
Compiling datafusion-proto v55.1.0 (/workspace/datafusion-branch/datafusion/proto)
Compiling datafusion-benchmarks v55.1.0 (/workspace/datafusion-branch/benchmarks)
Finished `bench` profile [optimized] target(s) in 8m 35s
Running benches/sql.rs (/workspace/datafusion-branch/target/release/deps/sql-2af2193982da74c5)
Gnuplot not found, using plotters backend
Benchmarking spill_views/q01_sort_string_1_distinct_repeated
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Collecting 10 samples in estimated 5.2638 s (60 iterations)
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Analyzing
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Collecting 10 samples in estimated 5.8173 s (60 iterations)
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Collecting 10 samples in estimated 5.8404 s (60 iterations)
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q04_sort_string_all_distinct_distinct
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Collecting 10 samples in estimated 6.3237 s (40 iterations)
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Analyzing
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Warming up for 3.0000 s
thread 'main' (15679) panicked at benchmarks/src/sql_benchmark_runner.rs:224:25:
Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted("Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\n FinalHashAggregateStream[2]#5656(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[1]#5655(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[0]#5665(can spill: true) consumed 27.1 MB, peak 27.1 MB,\n RepartitionExec[0]#5661(can spill: true) consumed 4.8 MB, peak 4.8 MB,\n RepartitionExec[2]#5663(can spill: true) consumed 2.4 MB, peak 2.4 MB.\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' (15679) panicked at benchmarks/benches/sql.rs:55:27:
failed to run SQL benchmarks: Execution("criterion benchmark failed: Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted(\"Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\\n FinalHashAggregateStream[2]#5656(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[1]#5655(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[0]#5665(can spill: true) consumed 27.1 MB, peak 27.1 MB,\\n RepartitionExec[0]#5661(can spill: true) consumed 4.8 MB, peak 4.8 MB,\\n RepartitionExec[2]#5663(can spill: true) consumed 2.4 MB, peak 2.4 MB.\\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)\")")
error: bench failed, to rerun pass `--bench sql`
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783384486
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783079245)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 137.74 ms β 134.85 ms β no change β
β Q2 β 139.32 ms β 137.69 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 203.11 ms β 198.18 ms β no change β
β Q5 β 233.73 ms β 225.75 ms β no change β
β Q6 β 252.44 ms β 251.17 ms β no change β
β Q7 β 695.52 ms β 682.11 ms β no change β
β Q8 β 489.87 ms β 506.53 ms β no change β
β Q9 β 541.80 ms β 564.84 ms β no change β
β Q10 β 1263.39 ms β 1260.91 ms β no change β
β Q11 β 378.02 ms β 420.69 ms β 1.11x slower β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4334.92ms β
β Total Time (spill-dedup-view-arrays) β 4382.71ms β
β Average Time (HEAD) β 433.49ms β
β Average Time (spill-dedup-view-arrays) β 438.27ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 9 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 137.74 / 138.83 Β±1.11 / 140.93 ms β 134.85 / 135.47 Β±0.75 / 136.73 ms β no change β
β Q2 β 139.32 / 140.15 Β±1.21 / 142.56 ms β 137.69 / 138.64 Β±0.66 / 139.34 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 203.11 / 206.30 Β±3.21 / 212.42 ms β 198.18 / 203.84 Β±4.64 / 209.70 ms β no change β
β Q5 β 233.73 / 235.66 Β±1.29 / 237.69 ms β 225.75 / 229.73 Β±2.69 / 232.41 ms β no change β
β Q6 β 252.44 / 255.92 Β±2.79 / 258.74 ms β 251.17 / 254.18 Β±1.89 / 256.87 ms β no change β
β Q7 β 695.52 / 714.98 Β±13.23 / 736.77 ms β 682.11 / 694.94 Β±17.08 / 726.52 ms β no change β
β Q8 β 489.87 / 500.93 Β±7.18 / 511.97 ms β 506.53 / 521.90 Β±8.50 / 530.55 ms β no change β
β Q9 β 541.80 / 555.67 Β±8.03 / 566.22 ms β 564.84 / 573.02 Β±6.45 / 583.42 ms β no change β
β Q10 β 1263.39 / 1275.00 Β±12.05 / 1297.37 ms β 1260.91 / 1298.74 Β±25.15 / 1328.53 ms β no change β
β Q11 β 378.02 / 385.16 Β±5.76 / 392.86 ms β 420.69 / 426.80 Β±6.44 / 438.18 ms β 1.11x slower β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4408.60ms β
β Total Time (spill-dedup-view-arrays) β 4477.26ms β
β Average Time (HEAD) β 440.86ms β
β Average Time (spill-dedup-view-arrays) β 447.73ms β
β Queries Faster β 0 β
β Queries Slower β 1 β
β Queries with No Change β 9 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `5b5bed9 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 174.4 MiB | 175.6 MiB | +0.7% |
| 2 | 219.9 MiB | 215.9 MiB | -1.8% |
| 3 | 552.2 MiB | 524.2 MiB | -5.1% |
| 4 | 263.6 MiB | 262.1 MiB | -0.6% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 353.1 MiB | 350.9 MiB | -0.6% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 499.4 MiB | 502.7 MiB | +0.7% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 523.0 MiB | 523.6 MiB | +0.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`5b5bed9 (merge-base)`) | 552.2 MiB | 1.2 GiB | 690.7 MiB | 2.3Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 524.2 MiB | 1.0 GiB | 545.4 MiB | 2.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 897.2 MiB |
| CPU user | 68.5s |
| CPU sys | 17.3s |
| Peak spill | 1.2 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.0 GiB |
| Avg memory | 788.9 MiB |
| CPU user | 66.0s |
| CPU sys | 16.4s |
| Peak spill | 972.8 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783410937
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783081490)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.02 ms β 136.61 ms β no change β
β Q2 β 136.72 ms β 137.07 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.96 ms β 199.16 ms β no change β
β Q5 β 228.52 ms β 229.93 ms β no change β
β Q6 β 253.65 ms β 252.12 ms β no change β
β Q7 β 677.96 ms β 675.70 ms β no change β
β Q8 β 474.34 ms β 488.26 ms β no change β
β Q9 β 515.96 ms β 529.68 ms β no change β
β Q10 β 1220.75 ms β 1222.73 ms β no change β
β Q11 β 369.51 ms β 372.46 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4211.38ms β
β Total Time (spill-dedup-view-arrays) β 4243.72ms β
β Average Time (HEAD) β 421.14ms β
β Average Time (spill-dedup-view-arrays) β 424.37ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.02 / 137.08 Β±1.56 / 139.75 ms β 136.61 / 138.41 Β±1.58 / 141.06 ms β no change β
β Q2 β 136.72 / 137.66 Β±1.00 / 139.28 ms β 137.07 / 137.87 Β±0.74 / 138.99 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.96 / 201.71 Β±2.88 / 207.05 ms β 199.16 / 205.26 Β±5.39 / 213.20 ms β no change β
β Q5 β 228.52 / 234.45 Β±4.65 / 242.44 ms β 229.93 / 231.83 Β±1.22 / 232.91 ms β no change β
β Q6 β 253.65 / 256.85 Β±1.91 / 259.21 ms β 252.12 / 253.82 Β±1.00 / 254.92 ms β no change β
β Q7 β 677.96 / 693.09 Β±8.72 / 704.50 ms β 675.70 / 691.12 Β±8.29 / 698.00 ms β no change β
β Q8 β 474.34 / 487.43 Β±7.90 / 498.92 ms β 488.26 / 492.48 Β±3.38 / 498.36 ms β no change β
β Q9 β 515.96 / 529.72 Β±10.09 / 543.00 ms β 529.68 / 534.97 Β±5.41 / 544.98 ms β no change β
β Q10 β 1220.75 / 1232.56 Β±10.76 / 1247.14 ms β 1222.73 / 1241.29 Β±11.92 / 1253.09 ms β no change β
β Q11 β 369.51 / 376.14 Β±4.91 / 383.34 ms β 372.46 / 377.24 Β±2.78 / 380.10 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4286.70ms β
β Total Time (spill-dedup-view-arrays) β 4304.29ms β
β Average Time (HEAD) β 428.67ms β
β Average Time (spill-dedup-view-arrays) β 430.43ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `5b5bed9 (merge-base)` | Changed: `5b5bed97f5fdbbb9a7026653fab4b21170a7a856`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.1 MiB | 175.6 MiB | +0.3% |
| 2 | 220.1 MiB | 220.6 MiB | +0.2% |
| 3 | 552.2 MiB | 514.3 MiB | -6.9% |
| 4 | 264.8 MiB | 266.1 MiB | +0.5% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 356.1 MiB | 356.1 MiB | +0.0% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 502.8 MiB | 497.9 MiB | -1.0% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 524.0 MiB | 525.3 MiB | +0.2% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`5b5bed9 (merge-base)`) | 552.2 MiB | 1.2 GiB | 649.1 MiB | 2.2Γ |
| sort_tpch | changed (`5b5bed97f5fdbbb9a7026653fab4b21170a7a856`) | 525.3 MiB | 1.1 GiB | 625.0 MiB | 2.2Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 861.8 MiB |
| CPU user | 66.5s |
| CPU sys | 15.9s |
| Peak spill | 1.1 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 764.8 MiB |
| CPU user | 63.2s |
| CPU sys | 15.2s |
| Peak spill | 971.5 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783909606
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783728693)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.19 ms β 136.45 ms β no change β
β Q2 β 136.50 ms β 136.90 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 200.99 ms β 199.82 ms β no change β
β Q5 β 230.10 ms β 230.75 ms β no change β
β Q6 β 254.15 ms β 253.53 ms β no change β
β Q7 β 686.62 ms β 674.88 ms β no change β
β Q8 β 475.37 ms β 488.19 ms β no change β
β Q9 β 523.93 ms β 538.07 ms β no change β
β Q10 β 1267.47 ms β 1248.09 ms β no change β
β Q11 β 387.09 ms β 376.15 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4297.42ms β
β Total Time (spill-dedup-view-arrays) β 4282.84ms β
β Average Time (HEAD) β 429.74ms β
β Average Time (spill-dedup-view-arrays) β 428.28ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.19 / 136.73 Β±1.76 / 139.96 ms β 136.45 / 137.78 Β±1.58 / 140.68 ms β no change β
β Q2 β 136.50 / 138.73 Β±1.70 / 140.98 ms β 136.90 / 138.28 Β±1.12 / 139.76 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 200.99 / 205.42 Β±3.75 / 209.68 ms β 199.82 / 201.64 Β±1.31 / 203.29 ms β no change β
β Q5 β 230.10 / 233.27 Β±2.54 / 237.53 ms β 230.75 / 234.46 Β±2.80 / 238.44 ms β no change β
β Q6 β 254.15 / 257.24 Β±1.97 / 259.35 ms β 253.53 / 258.79 Β±5.17 / 267.79 ms β no change β
β Q7 β 686.62 / 702.27 Β±10.80 / 717.74 ms β 674.88 / 699.41 Β±13.96 / 715.24 ms β no change β
β Q8 β 475.37 / 492.82 Β±11.69 / 508.90 ms β 488.19 / 495.31 Β±7.01 / 508.23 ms β no change β
β Q9 β 523.93 / 535.19 Β±7.92 / 544.36 ms β 538.07 / 548.00 Β±8.09 / 559.96 ms β no change β
β Q10 β 1267.47 / 1279.76 Β±11.20 / 1297.47 ms β 1248.09 / 1272.36 Β±17.68 / 1301.35 ms β no change β
β Q11 β 387.09 / 395.94 Β±7.25 / 407.28 ms β 376.15 / 379.29 Β±2.21 / 381.84 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4377.37ms β
β Total Time (spill-dedup-view-arrays) β 4365.31ms β
β Average Time (HEAD) β 437.74ms β
β Average Time (spill-dedup-view-arrays) β 436.53ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `5b5bed9 (merge-base)` | Changed: `5b5bed97f5fdbbb9a7026653fab4b21170a7a856`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 174.9 MiB | 175.3 MiB | +0.2% |
| 2 | 220.8 MiB | 220.8 MiB | +0.0% |
| 3 | 523.3 MiB | 552.2 MiB | +5.5% |
| 4 | 265.2 MiB | 265.5 MiB | +0.1% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 356.1 MiB | 355.7 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 497.9 MiB | +0.0% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 522.5 MiB | 522.7 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`5b5bed9 (merge-base)`) | 523.3 MiB | 1.1 GiB | 582.7 MiB | 2.1Γ |
| sort_tpch | changed (`5b5bed97f5fdbbb9a7026653fab4b21170a7a856`) | 552.2 MiB | 1.1 GiB | 559.8 MiB | 2.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 824.1 MiB |
| CPU user | 65.9s |
| CPU sys | 16.3s |
| Peak spill | 1.1 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 814.9 MiB |
| CPU user | 66.1s |
| CPU sys | 16.4s |
| Peak spill | 1.2 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783918616
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783733220)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783733220-2608-bgj7t 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "4"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783829693
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.98 ms β 5.81 ms β no change β
β QQuery 2 β 82.06 ms β 81.41 ms β no change β
β QQuery 3 β 30.25 ms β 29.20 ms β no change β
β QQuery 4 β 502.77 ms β 460.23 ms β +1.09x faster β
β QQuery 5 β 53.07 ms β 53.42 ms β no change β
β QQuery 6 β 37.62 ms β 37.57 ms β no change β
β QQuery 7 β 76.19 ms β 75.92 ms β no change β
β QQuery 8 β 37.83 ms β 37.50 ms β no change β
β QQuery 9 β 53.50 ms β 54.82 ms β no change β
β QQuery 10 β 63.93 ms β 63.54 ms β no change β
β QQuery 11 β 320.72 ms β 322.60 ms β no change β
β QQuery 12 β 31.02 ms β 31.62 ms β no change β
β QQuery 13 β 123.08 ms β 123.41 ms β no change β
β QQuery 14 β 413.86 ms β 402.64 ms β no change β
β QQuery 15 β 62.74 ms β 62.59 ms β no change β
β QQuery 16 β 7.07 ms β 7.05 ms β no change β
β QQuery 17 β 82.56 ms β 80.00 ms β no change β
β QQuery 18 β 113.23 ms β 107.38 ms β +1.05x faster β
β QQuery 19 β 44.31 ms β 43.00 ms β no change β
β QQuery 20 β 38.65 ms β 38.17 ms β no change β
β QQuery 21 β 18.38 ms β 18.19 ms β no change β
β QQuery 22 β 69.12 ms β 70.40 ms β no change β
β QQuery 23 β 341.51 ms β 339.20 ms β no change β
β QQuery 24 β 215.40 ms β 210.04 ms β no change β
β QQuery 25 β 114.78 ms β 115.31 ms β no change β
β QQuery 26 β 51.40 ms β 50.31 ms β no change β
β QQuery 27 β 6.97 ms β 6.70 ms β no change β
β QQuery 28 β 59.23 ms β 58.32 ms β no change β
β QQuery 29 β 98.28 ms β 97.54 ms β no change β
β QQuery 30 β 35.63 ms β 33.88 ms β no change β
β QQuery 31 β 115.19 ms β 113.63 ms β no change β
β QQuery 32 β 22.00 ms β 21.26 ms β no change β
β QQuery 33 β 38.82 ms β 38.99 ms β no change β
β QQuery 34 β 10.93 ms β 10.77 ms β no change β
β QQuery 35 β 78.73 ms β 78.31 ms β no change β
β QQuery 36 β 6.57 ms β 6.37 ms β no change β
β QQuery 37 β 7.60 ms β 7.47 ms β no change β
β QQuery 38 β 66.30 ms β 65.29 ms β no change β
β QQuery 39 β 77.48 ms β 78.11 ms β no change β
β QQuery 40 β 25.24 ms β 25.84 ms β no change β
β QQuery 41 β 11.93 ms β 11.96 ms β no change β
β QQuery 42 β 24.52 ms β 24.47 ms β no change β
β QQuery 43 β 5.65 ms β 5.76 ms β no change β
β QQuery 44 β 9.96 ms β 10.20 ms β no change β
β QQuery 45 β 45.58 ms β 41.85 ms β +1.09x faster β
β QQuery 46 β 13.17 ms β 12.68 ms β no change β
β QQuery 47 β 278.58 ms β 254.27 ms β +1.10x faster β
β QQuery 48 β 99.91 ms β 98.70 ms β no change β
β QQuery 49 β 73.57 ms β 72.22 ms β no change β
β QQuery 50 β 60.06 ms β 59.28 ms β no change β
β QQuery 51 β 95.60 ms β 92.54 ms β no change β
β QQuery 52 β 25.40 ms β 24.66 ms β no change β
β QQuery 53 β 30.24 ms β 29.50 ms β no change β
β QQuery 54 β 55.44 ms β 55.05 ms β no change β
β QQuery 55 β 24.54 ms β 23.98 ms β no change β
β QQuery 56 β 39.91 ms β 38.83 ms β no change β
β QQuery 57 β 175.28 ms β 170.60 ms β no change β
β QQuery 58 β 112.89 ms β 112.39 ms β no change β
β QQuery 59 β 119.21 ms β 119.69 ms β no change β
β QQuery 60 β 39.16 ms β 39.38 ms β no change β
β QQuery 61 β 11.79 ms β 11.85 ms β no change β
β QQuery 62 β 45.21 ms β 45.31 ms β no change β
β QQuery 63 β 29.46 ms β 29.52 ms β no change β
β QQuery 64 β 363.98 ms β 363.09 ms β no change β
β QQuery 65 β 132.09 ms β 134.85 ms β no change β
β QQuery 66 β 82.13 ms β 81.35 ms β no change β
β QQuery 67 β 275.05 ms β 266.15 ms β no change β
β QQuery 68 β 12.67 ms β 12.52 ms β no change β
β QQuery 69 β 58.66 ms β 58.43 ms β no change β
β QQuery 70 β 107.90 ms β 107.05 ms β no change β
β QQuery 71 β 36.73 ms β 36.20 ms β no change β
β QQuery 72 β 1819.99 ms β 1831.75 ms β no change β
β QQuery 73 β 10.99 ms β 10.53 ms β no change β
β QQuery 74 β 182.69 ms β 178.56 ms β no change β
β QQuery 75 β 146.18 ms β 144.32 ms β no change β
β QQuery 76 β 35.40 ms β 35.45 ms β no change β
β QQuery 77 β 62.26 ms β 62.39 ms β no change β
β QQuery 78 β 170.72 ms β 167.05 ms β no change β
β QQuery 79 β 70.82 ms β 67.81 ms β no change β
β QQuery 80 β 104.41 ms β 101.29 ms β no change β
β QQuery 81 β 28.46 ms β 27.45 ms β no change β
β QQuery 82 β 17.79 ms β 17.45 ms β no change β
β QQuery 83 β 35.61 ms β 35.25 ms β no change β
β QQuery 84 β 30.59 ms β 30.35 ms β no change β
β QQuery 85 β 106.82 ms β 107.15 ms β no change β
β QQuery 86 β 27.13 ms β 25.96 ms β no change β
β QQuery 87 β 67.66 ms β 65.03 ms β no change β
β QQuery 88 β 63.17 ms β 62.23 ms β no change β
β QQuery 89 β 36.95 ms β 35.97 ms β no change β
β QQuery 90 β 17.59 ms β 17.43 ms β no change β
β QQuery 91 β 45.89 ms β 45.95 ms β no change β
β QQuery 92 β 31.04 ms β 31.36 ms β no change β
β QQuery 93 β 52.82 ms β 51.51 ms β no change β
β QQuery 94 β 41.94 ms β 40.59 ms β no change β
β QQuery 95 β 84.53 ms β 83.24 ms β no change β
β QQuery 96 β 24.25 ms β 24.54 ms β no change β
β QQuery 97 β 53.21 ms β 52.81 ms β no change β
β QQuery 98 β 46.16 ms β 45.36 ms β no change β
β QQuery 99 β 67.81 ms β 68.88 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9619.15ms β
β Total Time (spill-dedup-view-arrays) β 9475.83ms β
β Average Time (HEAD) β 97.16ms β
β Average Time (spill-dedup-view-arrays) β 95.72ms β
β Queries Faster β 4 β
β Queries Slower β 0 β
β Queries with No Change β 95 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.98 / 6.62 Β±0.96 / 8.54 ms β 5.81 / 6.38 Β±0.95 / 8.28 ms β no change β
β QQuery 2 β 82.06 / 83.28 Β±0.70 / 84.02 ms β 81.41 / 81.94 Β±0.42 / 82.69 ms β no change β
β QQuery 3 β 30.25 / 30.54 Β±0.23 / 30.93 ms β 29.20 / 29.38 Β±0.14 / 29.56 ms β no change β
β QQuery 4 β 502.77 / 521.55 Β±16.68 / 551.57 ms β 460.23 / 477.69 Β±14.57 / 502.46 ms β +1.09x faster β
β QQuery 5 β 53.07 / 53.35 Β±0.18 / 53.56 ms β 53.42 / 53.65 Β±0.22 / 53.99 ms β no change β
β QQuery 6 β 37.62 / 37.93 Β±0.19 / 38.13 ms β 37.57 / 37.79 Β±0.26 / 38.31 ms β no change β
β QQuery 7 β 76.19 / 78.68 Β±3.91 / 86.45 ms β 75.92 / 76.89 Β±0.76 / 77.83 ms β no change β
β QQuery 8 β 37.83 / 38.39 Β±0.45 / 39.22 ms β 37.50 / 38.75 Β±2.24 / 43.22 ms β no change β
β QQuery 9 β 53.50 / 56.46 Β±1.77 / 59.03 ms β 54.82 / 57.16 Β±1.42 / 58.92 ms β no change β
β QQuery 10 β 63.93 / 64.44 Β±0.48 / 65.35 ms β 63.54 / 63.97 Β±0.25 / 64.30 ms β no change β
β QQuery 11 β 320.72 / 325.91 Β±4.02 / 331.13 ms β 322.60 / 328.46 Β±3.46 / 332.93 ms β no change β
β QQuery 12 β 31.02 / 31.49 Β±0.46 / 32.24 ms β 31.62 / 31.96 Β±0.24 / 32.36 ms β no change β
β QQuery 13 β 123.08 / 125.70 Β±1.87 / 127.88 ms β 123.41 / 124.70 Β±1.18 / 126.82 ms β no change β
β QQuery 14 β 413.86 / 418.11 Β±2.20 / 420.25 ms β 402.64 / 413.05 Β±8.16 / 424.14 ms β no change β
β QQuery 15 β 62.74 / 63.64 Β±0.73 / 64.62 ms β 62.59 / 63.51 Β±0.90 / 64.91 ms β no change β
β QQuery 16 β 7.07 / 7.37 Β±0.19 / 7.66 ms β 7.05 / 7.14 Β±0.08 / 7.28 ms β no change β
β QQuery 17 β 82.56 / 83.67 Β±1.08 / 85.20 ms β 80.00 / 82.48 Β±2.30 / 86.74 ms β no change β
β QQuery 18 β 113.23 / 114.48 Β±1.19 / 116.59 ms β 107.38 / 108.42 Β±0.82 / 109.65 ms β +1.06x faster β
β QQuery 19 β 44.31 / 44.48 Β±0.15 / 44.73 ms β 43.00 / 43.31 Β±0.29 / 43.80 ms β no change β
β QQuery 20 β 38.65 / 40.44 Β±1.05 / 41.69 ms β 38.17 / 39.40 Β±0.83 / 40.47 ms β no change β
β QQuery 21 β 18.38 / 18.48 Β±0.13 / 18.74 ms β 18.19 / 18.40 Β±0.26 / 18.90 ms β no change β
β QQuery 22 β 69.12 / 71.60 Β±1.29 / 72.86 ms β 70.40 / 71.78 Β±1.35 / 74.13 ms β no change β
β QQuery 23 β 341.51 / 348.15 Β±7.79 / 363.26 ms β 339.20 / 344.28 Β±6.00 / 355.98 ms β no change β
β QQuery 24 β 215.40 / 221.27 Β±4.86 / 227.42 ms β 210.04 / 216.86 Β±6.74 / 228.34 ms β no change β
β QQuery 25 β 114.78 / 117.13 Β±1.63 / 119.54 ms β 115.31 / 115.95 Β±0.89 / 117.71 ms β no change β
β QQuery 26 β 51.40 / 52.24 Β±0.85 / 53.81 ms β 50.31 / 52.37 Β±3.00 / 58.34 ms β no change β
β QQuery 27 β 6.97 / 7.07 Β±0.07 / 7.14 ms β 6.70 / 6.82 Β±0.12 / 7.04 ms β no change β
β QQuery 28 β 59.23 / 62.40 Β±1.61 / 63.46 ms β 58.32 / 61.16 Β±1.75 / 62.94 ms β no change β
β QQuery 29 β 98.28 / 103.11 Β±5.70 / 113.72 ms β 97.54 / 99.85 Β±2.52 / 103.31 ms β no change β
β QQuery 30 β 35.63 / 36.26 Β±0.42 / 36.92 ms β 33.88 / 34.53 Β±0.58 / 35.60 ms β no change β
β QQuery 31 β 115.19 / 115.62 Β±0.34 / 115.90 ms β 113.63 / 114.53 Β±0.96 / 116.39 ms β no change β
β QQuery 32 β 22.00 / 24.00 Β±3.38 / 30.75 ms β 21.26 / 21.78 Β±0.27 / 22.05 ms β +1.10x faster β
β QQuery 33 β 38.82 / 39.81 Β±0.81 / 41.11 ms β 38.99 / 41.07 Β±3.58 / 48.22 ms β no change β
β QQuery 34 β 10.93 / 11.51 Β±0.60 / 12.50 ms β 10.77 / 11.15 Β±0.45 / 12.00 ms β no change β
β QQuery 35 β 78.73 / 79.37 Β±0.52 / 80.27 ms β 78.31 / 78.87 Β±0.52 / 79.60 ms β no change β
β QQuery 36 β 6.57 / 6.71 Β±0.09 / 6.81 ms β 6.37 / 6.46 Β±0.09 / 6.60 ms β no change β
β QQuery 37 β 7.60 / 7.70 Β±0.06 / 7.77 ms β 7.47 / 7.54 Β±0.08 / 7.69 ms β no change β
β QQuery 38 β 66.30 / 67.38 Β±1.03 / 68.80 ms β 65.29 / 65.95 Β±0.40 / 66.36 ms β no change β
β QQuery 39 β 77.48 / 79.27 Β±1.47 / 81.53 ms β 78.11 / 79.69 Β±2.22 / 84.10 ms β no change β
β QQuery 40 β 25.24 / 25.45 Β±0.23 / 25.85 ms β 25.84 / 26.39 Β±0.31 / 26.69 ms β no change β
β QQuery 41 β 11.93 / 12.01 Β±0.06 / 12.08 ms β 11.96 / 12.23 Β±0.15 / 12.37 ms β no change β
β QQuery 42 β 24.52 / 24.66 Β±0.11 / 24.79 ms β 24.47 / 25.14 Β±0.58 / 25.92 ms β no change β
β QQuery 43 β 5.65 / 5.82 Β±0.13 / 5.94 ms β 5.76 / 5.84 Β±0.06 / 5.93 ms β no change β
β QQuery 44 β 9.96 / 10.09 Β±0.10 / 10.25 ms β 10.20 / 10.32 Β±0.08 / 10.40 ms β no change β
β QQuery 45 β 45.58 / 49.83 Β±3.80 / 54.64 ms β 41.85 / 43.66 Β±1.25 / 44.80 ms β +1.14x faster β
β QQuery 46 β 13.17 / 13.67 Β±0.35 / 14.13 ms β 12.68 / 13.41 Β±0.99 / 15.34 ms β no change β
β QQuery 47 β 278.58 / 284.68 Β±3.86 / 288.77 ms β 254.27 / 262.74 Β±5.65 / 268.62 ms β +1.08x faster β
β QQuery 48 β 99.91 / 104.11 Β±5.97 / 115.95 ms β 98.70 / 99.70 Β±1.10 / 101.81 ms β no change β
β QQuery 49 β 73.57 / 74.94 Β±1.77 / 78.11 ms β 72.22 / 75.42 Β±4.91 / 85.18 ms β no change β
β QQuery 50 β 60.06 / 60.70 Β±0.48 / 61.34 ms β 59.28 / 59.90 Β±0.64 / 61.10 ms β no change β
β QQuery 51 β 95.60 / 100.40 Β±5.17 / 110.04 ms β 92.54 / 94.15 Β±1.27 / 95.56 ms β +1.07x faster β
β QQuery 52 β 25.40 / 26.04 Β±0.60 / 27.15 ms β 24.66 / 25.67 Β±1.13 / 27.77 ms β no change β
β QQuery 53 β 30.24 / 30.50 Β±0.24 / 30.88 ms β 29.50 / 29.57 Β±0.05 / 29.65 ms β no change β
β QQuery 54 β 55.44 / 55.98 Β±0.55 / 56.93 ms β 55.05 / 55.90 Β±0.75 / 57.29 ms β no change β
β QQuery 55 β 24.54 / 24.98 Β±0.37 / 25.44 ms β 23.98 / 24.33 Β±0.30 / 24.73 ms β no change β
β QQuery 56 β 39.91 / 41.64 Β±1.57 / 43.89 ms β 38.83 / 39.69 Β±0.53 / 40.39 ms β no change β
β QQuery 57 β 175.28 / 177.79 Β±2.78 / 183.20 ms β 170.60 / 173.24 Β±1.72 / 175.32 ms β no change β
β QQuery 58 β 112.89 / 114.41 Β±1.70 / 117.66 ms β 112.39 / 113.53 Β±1.74 / 116.98 ms β no change β
β QQuery 59 β 119.21 / 120.92 Β±2.44 / 125.76 ms β 119.69 / 120.30 Β±0.33 / 120.64 ms β no change β
β QQuery 60 β 39.16 / 39.79 Β±0.34 / 40.15 ms β 39.38 / 40.23 Β±0.57 / 40.86 ms β no change β
β QQuery 61 β 11.79 / 11.95 Β±0.12 / 12.15 ms β 11.85 / 11.96 Β±0.11 / 12.17 ms β no change β
β QQuery 62 β 45.21 / 45.32 Β±0.08 / 45.42 ms β 45.31 / 46.08 Β±1.27 / 48.62 ms β no change β
β QQuery 63 β 29.46 / 29.86 Β±0.28 / 30.32 ms β 29.52 / 29.89 Β±0.30 / 30.29 ms β no change β
β QQuery 64 β 363.98 / 373.07 Β±10.08 / 391.58 ms β 363.09 / 368.41 Β±8.46 / 385.19 ms β no change β
β QQuery 65 β 132.09 / 138.52 Β±3.36 / 141.06 ms β 134.85 / 136.98 Β±1.55 / 138.94 ms β no change β
β QQuery 66 β 82.13 / 83.83 Β±1.15 / 85.61 ms β 81.35 / 82.68 Β±1.21 / 84.66 ms β no change β
β QQuery 67 β 275.05 / 287.08 Β±7.77 / 297.00 ms β 266.15 / 271.27 Β±4.98 / 278.67 ms β +1.06x faster β
β QQuery 68 β 12.67 / 13.04 Β±0.21 / 13.26 ms β 12.52 / 12.67 Β±0.13 / 12.84 ms β no change β
β QQuery 69 β 58.66 / 59.15 Β±0.54 / 59.97 ms β 58.43 / 59.60 Β±1.60 / 62.76 ms β no change β
β QQuery 70 β 107.90 / 114.04 Β±6.99 / 126.52 ms β 107.05 / 108.05 Β±0.94 / 109.30 ms β +1.06x faster β
β QQuery 71 β 36.73 / 37.18 Β±0.36 / 37.81 ms β 36.20 / 36.70 Β±0.44 / 37.47 ms β no change β
β QQuery 72 β 1819.99 / 1955.67 Β±74.67 / 2041.24 ms β 1831.75 / 1903.07 Β±57.55 / 2002.00 ms β no change β
β QQuery 73 β 10.99 / 11.45 Β±0.23 / 11.63 ms β 10.53 / 10.92 Β±0.42 / 11.70 ms β no change β
β QQuery 74 β 182.69 / 187.65 Β±3.52 / 192.01 ms β 178.56 / 180.96 Β±2.23 / 184.66 ms β no change β
β QQuery 75 β 146.18 / 147.72 Β±1.74 / 150.89 ms β 144.32 / 146.84 Β±3.44 / 153.56 ms β no change β
β QQuery 76 β 35.40 / 35.97 Β±0.40 / 36.48 ms β 35.45 / 35.99 Β±0.43 / 36.54 ms β no change β
β QQuery 77 β 62.26 / 64.06 Β±1.67 / 67.05 ms β 62.39 / 63.51 Β±1.65 / 66.78 ms β no change β
β QQuery 78 β 170.72 / 181.65 Β±13.05 / 206.71 ms β 167.05 / 170.26 Β±3.52 / 176.92 ms β +1.07x faster β
β QQuery 79 β 70.82 / 71.95 Β±0.90 / 73.54 ms β 67.81 / 68.60 Β±0.53 / 69.24 ms β no change β
β QQuery 80 β 104.41 / 105.55 Β±1.03 / 106.97 ms β 101.29 / 103.29 Β±2.52 / 108.27 ms β no change β
β QQuery 81 β 28.46 / 28.84 Β±0.25 / 29.23 ms β 27.45 / 28.20 Β±1.08 / 30.34 ms β no change β
β QQuery 82 β 17.79 / 17.95 Β±0.08 / 18.02 ms β 17.45 / 17.64 Β±0.20 / 17.89 ms β no change β
β QQuery 83 β 35.61 / 36.11 Β±0.41 / 36.81 ms β 35.25 / 35.46 Β±0.12 / 35.58 ms β no change β
β QQuery 84 β 30.59 / 31.17 Β±0.71 / 32.56 ms β 30.35 / 30.56 Β±0.16 / 30.79 ms β no change β
β QQuery 85 β 106.82 / 111.44 Β±6.24 / 123.26 ms β 107.15 / 110.26 Β±4.07 / 118.31 ms β no change β
β QQuery 86 β 27.13 / 27.65 Β±0.60 / 28.77 ms β 25.96 / 26.35 Β±0.34 / 26.95 ms β no change β
β QQuery 87 β 67.66 / 68.45 Β±0.64 / 69.50 ms β 65.03 / 66.29 Β±1.08 / 68.11 ms β no change β
β QQuery 88 β 63.17 / 63.65 Β±0.28 / 63.90 ms β 62.23 / 64.41 Β±3.64 / 71.65 ms β no change β
β QQuery 89 β 36.95 / 37.74 Β±0.75 / 38.97 ms β 35.97 / 36.41 Β±0.35 / 37.04 ms β no change β
β QQuery 90 β 17.59 / 17.77 Β±0.18 / 18.08 ms β 17.43 / 17.57 Β±0.09 / 17.68 ms β no change β
β QQuery 91 β 45.89 / 46.50 Β±0.35 / 46.91 ms β 45.95 / 46.42 Β±0.28 / 46.74 ms β no change β
β QQuery 92 β 31.04 / 32.19 Β±0.82 / 33.57 ms β 31.36 / 31.49 Β±0.12 / 31.71 ms β no change β
β QQuery 93 β 52.82 / 53.63 Β±0.60 / 54.42 ms β 51.51 / 52.81 Β±1.30 / 55.26 ms β no change β
β QQuery 94 β 41.94 / 45.45 Β±3.06 / 50.02 ms β 40.59 / 41.05 Β±0.51 / 42.04 ms β +1.11x faster β
β QQuery 95 β 84.53 / 86.20 Β±1.06 / 87.52 ms β 83.24 / 84.59 Β±1.28 / 86.90 ms β no change β
β QQuery 96 β 24.25 / 24.68 Β±0.24 / 24.95 ms β 24.54 / 24.69 Β±0.10 / 24.80 ms β no change β
β QQuery 97 β 53.21 / 53.99 Β±1.38 / 56.73 ms β 52.81 / 55.02 Β±2.58 / 59.97 ms β no change β
β QQuery 98 β 46.16 / 49.42 Β±3.69 / 56.27 ms β 45.36 / 46.67 Β±0.76 / 47.72 ms β +1.06x faster β
β QQuery 99 β 67.81 / 68.91 Β±0.69 / 69.85 ms β 68.88 / 69.15 Β±0.18 / 69.43 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9950.45ms β
β Total Time (spill-dedup-view-arrays) β 9699.24ms β
β Average Time (HEAD) β 100.51ms β
β Average Time (spill-dedup-view-arrays) β 97.97ms β
β Queries Faster β 11 β
β Queries Slower β 0 β
β Queries with No Change β 88 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 214.2s |
| CPU sys | 6.5s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.1 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 210.4s |
| CPU sys | 6.1s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783849854
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783728693)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783728693-2605-9zwmg 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783848964
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783725982-2603-z5gf4 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784044716
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783903325)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783903325-2610-7r8t2 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r3734842301
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -420,6 +420,20 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Garbage collect and deduplicate a string view before writing it out to disk
+///
+/// This is to avoid inflating disk usage and also to ensure that deduplication reduces memory pressure when reading back.
+fn gc_dedup_view<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> GenericByteViewArray<T> {
+ let mut builder = GenericByteViewBuilder::<T>::with_capacity(array.len())
Review Comment:
Every qualifying Utf8View and BinaryView now allocates a hash table sized for all rows and hashes every non-inline value. This is valuable for repeated values, but unique strings or large binary values get no disk reduction over gc() while paying extra CPU and peak memory during spilling.
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -1345,7 +1359,7 @@ mod tests {
let size_without_gc: usize = array_without_gc
.data_buffers()
.iter()
- .map(|buffer| buffer.capacity())
+ .map(|buffer| buffer.len())
Review Comment:
Keep coverage for both length and capacity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783727326
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784307832
run benchmark spill_views
env:
BENCH_QUERY: "4"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789150915
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789121802)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789121802-2623-4r587 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789153979
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789122907)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789122907-2624-87kn6 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870642804
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5870585310-2878-jgccc 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867809520
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867396562)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.50 109.7Β±0.91ms ? ?/sec 1.00 73.3Β±0.37ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.42 104.7Β±0.69ms ? ?/sec 1.00 73.7Β±0.42ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.40 103.3Β±1.03ms ? ?/sec 1.00 74.0Β±0.53ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 130.2Β±1.03ms ? ?/sec 1.00 130.1Β±0.96ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 135.7Β±4.61ms ? ?/sec 1.01 136.7Β±4.90ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69 (merge-base)`) | 102.6 MiB | 732.2 MiB | 629.6 MiB | 7.1Γ |
| spill_views | changed (`spill-dedup-view-arrays`) | 102.6 MiB | 774.8 MiB | 672.2 MiB | 7.6Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 380.2s |
| Peak memory | 732.2 MiB |
| Avg memory | 53.7 MiB |
| CPU user | 63.0s |
| CPU sys | 30.4s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 580.3s |
| Peak memory | 774.8 MiB |
| Avg memory | 41.6 MiB |
| CPU user | 75.7s |
| CPU sys | 24.3s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783310203
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783079245)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783079245-2591-6g6kr 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5786320092
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
resources:
cpu: "16"
memory: "128Gi"
arch: arm64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789389344
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789127530)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789127530-2629-tm28z 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789894565
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789851638)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789851638-2631-95lp8 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 7570366fd929daf9ced744bb8397686b50565b18 (7570366fd929daf9ced744bb8397686b50565b18) to 7570366fd929daf9ced744bb8397686b50565b18 [diff](https://github.com/apache/datafusion/compare/7570366fd929daf9ced744bb8397686b50565b18..7570366fd929daf9ced744bb8397686b50565b18)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
changed:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783120618
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783075232-2584-7q4h9 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783138029
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783075232-2585-knzkj 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783146975
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783076971-2588-vqk5q 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784274841
Benchmark for [this request](https://github.com/apache/datafusion/pull/23565#issuecomment-5783904761) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `spill_views`
<details><summary>Runner log (last 40 lines)</summary>
```
Compiling datafusion-datasource-json v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-json)
Compiling datafusion-datasource-csv v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-csv)
Compiling datafusion-datasource-arrow v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-arrow)
Compiling datafusion-datasource-parquet v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-parquet)
Compiling datafusion-physical-optimizer v55.1.0 (/workspace/datafusion-branch/datafusion/physical-optimizer)
Compiling datafusion-catalog-listing v55.1.0 (/workspace/datafusion-branch/datafusion/catalog-listing)
Compiling datafusion-functions-table v55.1.0 (/workspace/datafusion-branch/datafusion/functions-table)
Compiling datafusion v55.1.0 (/workspace/datafusion-branch/datafusion/core)
Compiling datafusion-proto v55.1.0 (/workspace/datafusion-branch/datafusion/proto)
Compiling datafusion-benchmarks v55.1.0 (/workspace/datafusion-branch/benchmarks)
Finished `bench` profile [optimized] target(s) in 5m 29s
Running benches/sql.rs (/workspace/datafusion-branch/target/release/deps/sql-2af2193982da74c5)
Gnuplot not found, using plotters backend
Benchmarking spill_views/q01_sort_string_1_distinct_repeated
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Collecting 10 samples in estimated 5.3885 s (60 iterations)
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Analyzing
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Collecting 10 samples in estimated 5.9061 s (60 iterations)
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Collecting 10 samples in estimated 5.9228 s (60 iterations)
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q04_sort_string_all_distinct_distinct
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Collecting 10 samples in estimated 5.4564 s (30 iterations)
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Analyzing
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Warming up for 3.0000 s
thread 'main' (15085) panicked at benchmarks/src/sql_benchmark_runner.rs:224:25:
Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted("Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\n FinalHashAggregateStream[1]#5550(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[2]#5551(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[0]#5559(can spill: true) consumed 27.1 MB, peak 27.1 MB,\n RepartitionExec[0]#5556(can spill: true) consumed 4.8 MB, peak 4.8 MB,\n RepartitionExec[2]#5558(can spill: true) consumed 2.4 MB, peak 2.4 MB.\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' (15085) panicked at benchmarks/benches/sql.rs:55:27:
failed to run SQL benchmarks: Execution("criterion benchmark failed: Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted(\"Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\\n FinalHashAggregateStream[1]#5550(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[2]#5551(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[0]#5559(can spill: true) consumed 27.1 MB, peak 27.1 MB,\\n RepartitionExec[0]#5556(can spill: true) consumed 4.8 MB, peak 4.8 MB,\\n RepartitionExec[2]#5558(can spill: true) consumed 2.4 MB, peak 2.4 MB.\\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)\")")
error: bench failed, to rerun pass `--bench sql`
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784279765
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783906160)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 104.2Β±0.96ms ? ?/sec 1.00 103.7Β±0.87ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 99.3Β±0.55ms ? ?/sec 1.00 99.5Β±0.56ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 98.1Β±0.48ms ? ?/sec 1.00 97.8Β±0.57ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 119.9Β±0.70ms ? ?/sec 1.00 119.6Β±0.74ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.02 133.8Β±2.64ms ? ?/sec 1.00 131.5Β±4.37ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 365.3s |
| Peak memory | 708.4 MiB |
| Avg memory | 55.7 MiB |
| CPU user | 64.7s |
| CPU sys | 30.4s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 365.2s |
| Peak memory | 761.8 MiB |
| Avg memory | 59.5 MiB |
| CPU user | 65.0s |
| CPU sys | 30.4s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783922345
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783731694)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783731694-2607-s77gf 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863142174
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110116)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863110116-2858-b585r 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863566906
## Benchmark bot results after the heuristics commit (head `7effeab5c`, base `ed43e69fd`)
I rebased onto `main` and changed the compaction to address the regressions from the last round:
- Deduplication samples the first 256 non-inline values. If they contain almost no repeats, it falls back to plain `gc()`, so all-distinct data no longer pays for hashing.
- Deduplication now uses a hash table over the views directly, instead of rebuilding the array through `GenericByteViewBuilder`.
A/B = main vs this PR, A/A = main vs main. Ratios are branch / base, so <1.00 is faster with this PR.
### spill_views
"Default limits" are the suite's own limits (40M for q01βq03, 96M for q04βq05).
| Query | A/B run 1 (default limits) | A/B run 2 (default limits) | A/A (default limits) | A/B (q05 at 128M) | A/A (q05 at 128M) |
| --- | --- | --- | --- | --- | --- |
| q01 sort, 1 distinct value | 108.0 β 84.3 ms (**0.78x**) | 107.8 β 85.4 ms (**0.79x**) | 0.99x | 105.0 β 83.5 ms (**0.80x**) | 1.01x |
| q02 sort, 1000 distinct values | 102.2 β 94.4 ms (**0.92x**) | 103.3 β 95.5 ms (**0.92x**) | 0.99x | 101.6 β 93.0 ms (**0.92x**) | 1.01x |
| q03 sort, BinaryView, 1000 distinct | 100.4 β 94.2 ms (**0.94x**) | 101.8 β 95.2 ms (**0.94x**) | 0.98x | 99.2 β 93.3 ms (**0.94x**) | 1.01x |
| q04 sort, all distinct | 1.00x | 1.02x | 0.99x | 1.00x | 1.01x |
| q05 GROUP BY, all distinct | 0.99x | 1.01x | 0.98x | 1.01x | 1.00x |
q05 now passes at the default 96M limit (2 of 2 runs; it previously failed on the branch side in 5 of 5).
### sort_tpch, 512M limit, 4 partitions
Q1βQ7 and Q10 show no change; Q3 fails on both sides, as before.
| Query | A/B run 1 | A/B run 2 | A/A |
| --- | --- | --- | --- |
| Q8 (`l_comment`) | 1.01x | 1.00x | 0.99x |
| Q9 (`l_comment`) | 1.00x | 1.00x | 1.00x |
| Q11 (`l_comment`) | 1.01x | 1.00x | 0.99x |
| Total | 1.00x | 1.00x | 0.99x |
### Compared with the previous round
| | Before | Now |
| --- | --- | --- |
| q01 | 0.84β0.86x | **0.78β0.80x** |
| q02 / q03 | ~1.00x | **0.92x / 0.94x** |
| q04 | 1.38β1.45x | 1.00β1.02x |
| q05 at 96M | OOM on the branch side, 5 of 5 runs | passes |
| sort_tpch Q11 | 1.08β1.11x | 1.00β1.01x |
The repeated-value queries are faster, and the all-distinct and `sort_tpch` cases are now within A/A noise. The reduction in spilled bytes is covered by `test_gc_copies_repeated_values_once` (the bot's peak-spill figure is sampled once per second and too coarse to show it).
@adriangb @kumarUjjawal could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867449904
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867397520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5867397520-2875-8xbcw 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867446600
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867396562)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5867396562-2873-9fsjg 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867838821
## Results for 9c2f8f19f (keep shared buffers instead of deduplicating values)
9c2f8f19f replaces the value dedup with a check on the buffers: merge repeated entries of the same buffer, then run `gc()` only if it writes less than the merged buffers hold. No values are hashed or sampled. Details are in the commit message.
### The issue's repro
1M rows, 1000 distinct labels, `-m 64M`, default partitions. `ORDER BY id DESC`, because `main` now skips the sort for `ORDER BY id` on this file. Three runs each, identical.
| Build | Spilled bytes |
| --- | --- |
| `main` | 83.2 MB |
| previous version (7effeab5c) | 83.2 MB |
| 9c2f8f19f | **31.0 MB** |
The previous version did not reduce this at all: sorted by `id`, the labels come out in cycling order, so the sample saw 256 distinct values and fell back to `gc()` in every batch.
### Multi-level merge: the repro at 10M rows, 1 partition
Median of 5 interleaved runs. Spilled rows above 10M are rows re-spilled by extra merge passes.
| Limit | Spilled: main β 9c2f8f19f | Spilled rows | Time |
| --- | --- | --- | --- |
| 24M | 1959 β 1152 MB | 23.5M β 20.0M | 832 β 751 ms |
| 32M | 1477 β 334 MB | 17.8M β **10.0M** (one pass) | 715 β **490 ms** |
| 48M | 832 β 353 MB | 10.0M β 10.0M | 571 β 501 ms |
| 64M | 832 β 371 MB | 10.0M β 10.0M | 574 β 494 ms |
| 128M | 832 β 444 MB | 10.0M β 10.0M | 614 β 578 ms |
Smaller spilled batches lower the per-file reservation in the multi-level merge, so more files fit in each pass. At 32M the merge needs one pass instead of about 1.8.
### spill_views and mixed data, local
`datafusion-cli`, 1M rows, 4 partitions, 15 rounds with the build order shuffled each round. Ratios are the median per-round time over `main`. "N% repeated" sets N% of the rows to one value, with the rest distinct.
| Case | Spilled: main / previous / 9c2f8f19f | previous / main | 9c2f8f19f / main |
| --- | --- | --- | --- |
| q01 sort, 1 distinct value | 84.2 / 23.2 / 23.8 MB | 0.97x | **0.85x** |
| q02 sort, 1000 distinct values | 84.2 / 30.6 / 31.1 MB | 1.05x | **0.88x** |
| q03 sort, BinaryView | 84.2 / 30.6 / 31.1 MB | 1.01x | **0.85x** |
| q04 sort, all distinct | 84.2 / 84.2 / 84.2 MB | 1.02x | 1.00x |
| q05 GROUP BY, all distinct | 84.2 / 84.2 / 84.2 MB | 0.94x | 1.01x |
| 5% repeated | 84.2 / 81.1 / 84.2 MB | **1.53x** | 1.04x |
| 20% repeated | 84.2 / 72.0 / 84.2 MB | **1.56x** | 0.99x |
| 50% repeated | 84.2 / 53.7 / 84.2 MB | **1.48x** | 1.04x |
| 80% repeated | 84.2 / 35.4 / 84.2 MB | **1.31x** | 1.00x |
| 1000 labels in cycling order | 84.2 / **84.2** / 31.1 MB | 1.03x | **0.79x** |
- Shared-dictionary data spills 2.7β3.6x less and runs 12β21% faster.
- Data without shared buffers takes the same `gc()` path as `main`: the same bytes, and times within noise (slower than `main` in 6β9 of 15 rounds).
- The trade-off: repeated values stored as separate copies (the 50% and 80% cases) are no longer deduplicated. The previous version spilled 36β58% less there, at 1.31β1.48x the time.
### Benchmark bot
Two A/B runs of 9c2f8f19f against (merge-base), , 4 partitions. The A/A (main vs main) runs are from the previous round, on the same merge-base. Ratios are branch / base, mean times.
**spill_views** (each query sets its own limit: 40M for q01βq03, 96M for q04βq05)
| Query | A/B run 1 | A/B run 2 | A/A (previous round) | Previous version (7effeab5c) |
| --- | --- | --- | --- | --- |
| q01 sort, 1 distinct value | 109.7 β 73.3 ms (**0.67x**) | 105.4 β 71.5 ms (**0.68x**) | 0.99x | 0.78β0.79x |
| q02 sort, 1000 distinct values | 104.7 β 73.7 ms (**0.70x**) | 100.5 β 71.5 ms (**0.71x**) | 0.99x | 0.92x |
| q03 sort, BinaryView | 103.3 β 74.0 ms (**0.72x**) | 100.2 β 71.8 ms (**0.72x**) | 0.98x | 0.94x |
| q04 sort, all distinct | 1.00x | 0.99x | 0.99x | 1.00β1.02x |
| q05 GROUP BY, all distinct | 1.01x | 0.97x | 0.98x | 0.99β1.01x |
q05 passed at the default 96M limit in both runs.
**sort_tpch** (SF1, 512M, 4 partitions): no query changes beyond A/A noise. Q3 fails on both sides, as before.
| Query | A/B run 1 | A/B run 2 | A/A (previous round) |
| --- | --- | --- | --- |
| Q8 () | 509 β 488 ms (0.96x) | 487 β 480 ms (0.98x) | 0.99x |
| Q9 () | 536 β 540 ms (1.01x) | 529 β 527 ms (1.00x) | 1.00x |
| Q11 () | 379 β 371 ms (0.98x) | 377 β 376 ms (1.00x) | 0.99x |
| Total | 4324 β 4254 ms (0.98x) | 4290 β 4267 ms (0.99x) | 0.99x |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870729300
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.50 ms β 6.02 ms β 1.09x slower β
β QQuery 2 β 81.22 ms β 80.35 ms β no change β
β QQuery 3 β 29.10 ms β 29.25 ms β no change β
β QQuery 4 β 481.86 ms β 459.87 ms β no change β
β QQuery 5 β 51.43 ms β 51.54 ms β no change β
β QQuery 6 β 36.20 ms β 35.91 ms β no change β
β QQuery 7 β 73.66 ms β 74.97 ms β no change β
β QQuery 8 β 36.52 ms β 36.24 ms β no change β
β QQuery 9 β 52.32 ms β 54.10 ms β no change β
β QQuery 10 β 62.20 ms β 61.96 ms β no change β
β QQuery 11 β 284.47 ms β 296.00 ms β no change β
β QQuery 12 β 28.94 ms β 29.68 ms β no change β
β QQuery 13 β 117.28 ms β 117.14 ms β no change β
β QQuery 14 β 394.81 ms β 396.61 ms β no change β
β QQuery 15 β 55.51 ms β 54.74 ms β no change β
β QQuery 16 β 6.56 ms β 7.06 ms β 1.08x slower β
β QQuery 17 β 80.25 ms β 79.48 ms β no change β
β QQuery 18 β 106.41 ms β 104.68 ms β no change β
β QQuery 19 β 41.68 ms β 41.75 ms β no change β
β QQuery 20 β 36.39 ms β 36.12 ms β no change β
β QQuery 21 β 17.05 ms β 17.54 ms β no change β
β QQuery 22 β 65.88 ms β 67.34 ms β no change β
β QQuery 23 β 400.63 ms β 413.60 ms β no change β
β QQuery 24 β 195.96 ms β 195.02 ms β no change β
β QQuery 25 β 107.36 ms β 108.26 ms β no change β
β QQuery 26 β 48.45 ms β 48.16 ms β no change β
β QQuery 27 β 6.29 ms β 6.49 ms β no change β
β QQuery 28 β 57.01 ms β 60.14 ms β 1.06x slower β
β QQuery 29 β 94.79 ms β 94.44 ms β no change β
β QQuery 30 β 33.95 ms β 32.79 ms β no change β
β QQuery 31 β 111.14 ms β 109.41 ms β no change β
β QQuery 32 β 20.20 ms β 20.90 ms β no change β
β QQuery 33 β 37.60 ms β 37.80 ms β no change β
β QQuery 34 β 10.36 ms β 10.05 ms β no change β
β QQuery 35 β 74.25 ms β 71.19 ms β no change β
β QQuery 36 β 6.15 ms β 5.69 ms β +1.08x faster β
β QQuery 37 β 6.76 ms β 6.86 ms β no change β
β QQuery 38 β 60.27 ms β 61.55 ms β no change β
β QQuery 39 β 73.69 ms β 73.56 ms β no change β
β QQuery 40 β 23.66 ms β 24.07 ms β no change β
β QQuery 41 β 11.80 ms β 11.91 ms β no change β
β QQuery 42 β 23.95 ms β 23.54 ms β no change β
β QQuery 43 β 4.67 ms β 4.97 ms β 1.06x slower β
β QQuery 44 β 8.79 ms β 8.64 ms β no change β
β QQuery 45 β 38.18 ms β 37.00 ms β no change β
β QQuery 46 β 11.48 ms β 11.81 ms β no change β
β QQuery 47 β 224.56 ms β 225.39 ms β no change β
β QQuery 48 β 94.26 ms β 94.82 ms β no change β
β QQuery 49 β 70.90 ms β 71.07 ms β no change β
β QQuery 50 β 58.24 ms β 57.83 ms β no change β
β QQuery 51 β 93.81 ms β 95.74 ms β no change β
β QQuery 52 β 23.97 ms β 24.00 ms β no change β
β QQuery 53 β 29.60 ms β 29.67 ms β no change β
β QQuery 54 β 53.73 ms β 54.85 ms β no change β
β QQuery 55 β 23.23 ms β 23.97 ms β no change β
β QQuery 56 β 39.15 ms β 38.84 ms β no change β
β QQuery 57 β 168.79 ms β 169.39 ms β no change β
β QQuery 58 β 108.74 ms β 108.63 ms β no change β
β QQuery 59 β 114.85 ms β 116.34 ms β no change β
β QQuery 60 β 38.22 ms β 39.32 ms β no change β
β QQuery 61 β 11.24 ms β 12.08 ms β 1.08x slower β
β QQuery 62 β 44.37 ms β 43.96 ms β no change β
β QQuery 63 β 29.80 ms β 29.85 ms β no change β
β QQuery 64 β 361.01 ms β 360.95 ms β no change β
β QQuery 65 β 132.37 ms β 131.45 ms β no change β
β QQuery 66 β 77.89 ms β 77.38 ms β no change β
β QQuery 67 β 294.00 ms β 296.14 ms β no change β
β QQuery 68 β 12.05 ms β 12.19 ms β no change β
β QQuery 69 β 56.64 ms β 56.48 ms β no change β
β QQuery 70 β 111.11 ms β 109.22 ms β no change β
β QQuery 71 β 34.98 ms β 35.48 ms β no change β
β QQuery 72 β 1696.97 ms β 1716.93 ms β no change β
β QQuery 73 β 9.76 ms β 10.09 ms β no change β
β QQuery 74 β 160.42 ms β 165.08 ms β no change β
β QQuery 75 β 142.08 ms β 142.08 ms β no change β
β QQuery 76 β 33.96 ms β 35.14 ms β no change β
β QQuery 77 β 61.57 ms β 59.98 ms β no change β
β QQuery 78 β 165.59 ms β 169.00 ms β no change β
β QQuery 79 β 67.53 ms β 66.81 ms β no change β
β QQuery 80 β 96.52 ms β 95.61 ms β no change β
β QQuery 81 β 25.89 ms β 26.91 ms β no change β
β QQuery 82 β 17.12 ms β 16.95 ms β no change β
β QQuery 83 β 33.53 ms β 33.32 ms β no change β
β QQuery 84 β 29.94 ms β 29.54 ms β no change β
β QQuery 85 β 102.46 ms β 103.10 ms β no change β
β QQuery 86 β 25.97 ms β 25.51 ms β no change β
β QQuery 87 β 61.61 ms β 60.43 ms β no change β
β QQuery 88 β 62.30 ms β 61.62 ms β no change β
β QQuery 89 β 35.51 ms β 34.88 ms β no change β
β QQuery 90 β 17.15 ms β 17.21 ms β no change β
β QQuery 91 β 45.64 ms β 44.49 ms β no change β
β QQuery 92 β 30.41 ms β 29.44 ms β no change β
β QQuery 93 β 49.53 ms β 49.29 ms β no change β
β QQuery 94 β 38.63 ms β 38.39 ms β no change β
β QQuery 95 β 81.58 ms β 80.01 ms β no change β
β QQuery 96 β 24.16 ms β 23.51 ms β no change β
β QQuery 97 β 51.00 ms β 49.94 ms β no change β
β QQuery 98 β 43.48 ms β 42.63 ms β no change β
β QQuery 99 β 66.89 ms β 65.59 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9197.36ms β
β Total Time (spill-dedup-view-arrays) β 9224.69ms β
β Average Time (HEAD) β 92.90ms β
β Average Time (spill-dedup-view-arrays) β 93.18ms β
β Queries Faster β 1 β
β Queries Slower β 5 β
β Queries with No Change β 93 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.50 / 6.04 Β±1.02 / 8.08 ms β 6.02 / 6.56 Β±0.92 / 8.40 ms β 1.09x slower β
β QQuery 2 β 81.22 / 81.56 Β±0.27 / 82.02 ms β 80.35 / 81.16 Β±0.56 / 82.06 ms β no change β
β QQuery 3 β 29.10 / 29.38 Β±0.17 / 29.64 ms β 29.25 / 29.49 Β±0.15 / 29.69 ms β no change β
β QQuery 4 β 481.86 / 490.09 Β±6.25 / 498.56 ms β 459.87 / 481.16 Β±10.72 / 488.57 ms β no change β
β QQuery 5 β 51.43 / 52.26 Β±0.70 / 53.56 ms β 51.54 / 54.48 Β±4.16 / 62.66 ms β no change β
β QQuery 6 β 36.20 / 36.85 Β±0.48 / 37.58 ms β 35.91 / 36.85 Β±0.54 / 37.45 ms β no change β
β QQuery 7 β 73.66 / 74.82 Β±0.92 / 75.89 ms β 74.97 / 75.35 Β±0.39 / 76.07 ms β no change β
β QQuery 8 β 36.52 / 37.88 Β±2.18 / 42.21 ms β 36.24 / 38.00 Β±2.60 / 43.16 ms β no change β
β QQuery 9 β 52.32 / 54.65 Β±1.83 / 57.78 ms β 54.10 / 57.07 Β±2.60 / 60.85 ms β no change β
β QQuery 10 β 62.20 / 62.70 Β±0.51 / 63.67 ms β 61.96 / 62.22 Β±0.23 / 62.53 ms β no change β
β QQuery 11 β 284.47 / 294.31 Β±7.52 / 304.12 ms β 296.00 / 305.55 Β±4.83 / 309.18 ms β no change β
β QQuery 12 β 28.94 / 29.46 Β±0.29 / 29.71 ms β 29.68 / 30.09 Β±0.35 / 30.61 ms β no change β
β QQuery 13 β 117.28 / 117.83 Β±0.56 / 118.64 ms β 117.14 / 118.56 Β±0.80 / 119.61 ms β no change β
β QQuery 14 β 394.81 / 402.64 Β±5.38 / 410.03 ms β 396.61 / 400.09 Β±3.57 / 406.80 ms β no change β
β QQuery 15 β 55.51 / 57.11 Β±1.61 / 59.55 ms β 54.74 / 57.99 Β±2.94 / 63.24 ms β no change β
β QQuery 16 β 6.56 / 6.91 Β±0.18 / 7.05 ms β 7.06 / 7.16 Β±0.12 / 7.37 ms β no change β
β QQuery 17 β 80.25 / 80.87 Β±0.78 / 82.37 ms β 79.48 / 81.06 Β±1.16 / 82.41 ms β no change β
β QQuery 18 β 106.41 / 107.08 Β±0.60 / 108.16 ms β 104.68 / 108.08 Β±2.01 / 110.26 ms β no change β
β QQuery 19 β 41.68 / 41.88 Β±0.15 / 42.08 ms β 41.75 / 42.19 Β±0.35 / 42.80 ms β no change β
β QQuery 20 β 36.39 / 36.71 Β±0.19 / 36.90 ms β 36.12 / 36.81 Β±0.42 / 37.30 ms β no change β
β QQuery 21 β 17.05 / 17.44 Β±0.36 / 18.00 ms β 17.54 / 17.64 Β±0.08 / 17.74 ms β no change β
β QQuery 22 β 65.88 / 67.43 Β±1.05 / 68.43 ms β 67.34 / 68.45 Β±0.86 / 69.84 ms β no change β
β QQuery 23 β 400.63 / 410.87 Β±7.56 / 420.58 ms β 413.60 / 418.06 Β±2.91 / 421.42 ms β no change β
β QQuery 24 β 195.96 / 201.04 Β±4.33 / 208.32 ms β 195.02 / 200.71 Β±4.64 / 204.97 ms β no change β
β QQuery 25 β 107.36 / 109.59 Β±1.41 / 111.57 ms β 108.26 / 108.70 Β±0.33 / 109.09 ms β no change β
β QQuery 26 β 48.45 / 49.49 Β±0.92 / 51.05 ms β 48.16 / 50.00 Β±1.78 / 53.40 ms β no change β
β QQuery 27 β 6.29 / 6.51 Β±0.21 / 6.89 ms β 6.49 / 6.65 Β±0.16 / 6.87 ms β no change β
β QQuery 28 β 57.01 / 59.51 Β±2.15 / 62.26 ms β 60.14 / 60.75 Β±0.40 / 61.34 ms β no change β
β QQuery 29 β 94.79 / 98.04 Β±2.44 / 101.86 ms β 94.44 / 96.61 Β±1.73 / 98.59 ms β no change β
β QQuery 30 β 33.95 / 34.95 Β±1.20 / 37.27 ms β 32.79 / 34.34 Β±2.24 / 38.79 ms β no change β
β QQuery 31 β 111.14 / 112.47 Β±1.07 / 114.07 ms β 109.41 / 111.11 Β±1.10 / 112.42 ms β no change β
β QQuery 32 β 20.20 / 20.75 Β±0.28 / 20.92 ms β 20.90 / 21.13 Β±0.22 / 21.45 ms β no change β
β QQuery 33 β 37.60 / 37.97 Β±0.23 / 38.21 ms β 37.80 / 38.89 Β±0.71 / 39.70 ms β no change β
β QQuery 34 β 10.36 / 10.60 Β±0.18 / 10.77 ms β 10.05 / 10.48 Β±0.43 / 11.12 ms β no change β
β QQuery 35 β 74.25 / 76.35 Β±1.72 / 78.87 ms β 71.19 / 72.99 Β±1.76 / 75.44 ms β no change β
β QQuery 36 β 6.15 / 6.34 Β±0.19 / 6.67 ms β 5.69 / 5.94 Β±0.27 / 6.30 ms β +1.07x faster β
β QQuery 37 β 6.76 / 6.90 Β±0.08 / 7.00 ms β 6.86 / 7.04 Β±0.12 / 7.20 ms β no change β
β QQuery 38 β 60.27 / 62.66 Β±1.93 / 65.12 ms β 61.55 / 62.87 Β±0.71 / 63.70 ms β no change β
β QQuery 39 β 73.69 / 74.95 Β±0.95 / 76.07 ms β 73.56 / 76.22 Β±2.10 / 79.73 ms β no change β
β QQuery 40 β 23.66 / 24.22 Β±0.49 / 25.11 ms β 24.07 / 24.63 Β±0.58 / 25.74 ms β no change β
β QQuery 41 β 11.80 / 11.93 Β±0.08 / 12.03 ms β 11.91 / 12.06 Β±0.09 / 12.20 ms β no change β
β QQuery 42 β 23.95 / 24.39 Β±0.38 / 25.05 ms β 23.54 / 23.77 Β±0.30 / 24.36 ms β no change β
β QQuery 43 β 4.67 / 4.89 Β±0.28 / 5.41 ms β 4.97 / 5.15 Β±0.19 / 5.45 ms β 1.05x slower β
β QQuery 44 β 8.79 / 8.92 Β±0.13 / 9.16 ms β 8.64 / 9.10 Β±0.30 / 9.51 ms β no change β
β QQuery 45 β 38.18 / 40.79 Β±1.74 / 42.76 ms β 37.00 / 38.72 Β±1.32 / 40.63 ms β +1.05x faster β
β QQuery 46 β 11.48 / 12.34 Β±0.52 / 12.91 ms β 11.81 / 11.98 Β±0.12 / 12.18 ms β no change β
β QQuery 47 β 224.56 / 227.90 Β±2.78 / 232.84 ms β 225.39 / 228.21 Β±2.75 / 232.24 ms β no change β
β QQuery 48 β 94.26 / 95.29 Β±0.62 / 96.05 ms β 94.82 / 95.98 Β±0.83 / 97.20 ms β no change β
β QQuery 49 β 70.90 / 72.50 Β±1.83 / 75.98 ms β 71.07 / 71.78 Β±0.73 / 73.16 ms β no change β
β QQuery 50 β 58.24 / 58.51 Β±0.21 / 58.84 ms β 57.83 / 59.80 Β±2.22 / 63.69 ms β no change β
β QQuery 51 β 93.81 / 96.61 Β±1.95 / 99.43 ms β 95.74 / 97.13 Β±1.38 / 98.83 ms β no change β
β QQuery 52 β 23.97 / 25.53 Β±1.69 / 28.81 ms β 24.00 / 24.35 Β±0.26 / 24.64 ms β no change β
β QQuery 53 β 29.60 / 29.90 Β±0.23 / 30.23 ms β 29.67 / 29.73 Β±0.05 / 29.81 ms β no change β
β QQuery 54 β 53.73 / 54.16 Β±0.29 / 54.46 ms β 54.85 / 55.62 Β±0.67 / 56.49 ms β no change β
β QQuery 55 β 23.23 / 23.68 Β±0.35 / 24.31 ms β 23.97 / 24.09 Β±0.12 / 24.31 ms β no change β
β QQuery 56 β 39.15 / 39.52 Β±0.27 / 39.97 ms β 38.84 / 39.52 Β±0.47 / 40.05 ms β no change β
β QQuery 57 β 168.79 / 170.07 Β±1.80 / 173.45 ms β 169.39 / 171.96 Β±2.20 / 175.46 ms β no change β
β QQuery 58 β 108.74 / 111.00 Β±2.27 / 114.90 ms β 108.63 / 109.20 Β±0.42 / 109.71 ms β no change β
β QQuery 59 β 114.85 / 116.06 Β±0.83 / 117.39 ms β 116.34 / 117.53 Β±1.38 / 120.12 ms β no change β
β QQuery 60 β 38.22 / 38.60 Β±0.37 / 39.29 ms β 39.32 / 39.90 Β±0.44 / 40.65 ms β no change β
β QQuery 61 β 11.24 / 12.47 Β±1.71 / 15.83 ms β 12.08 / 12.15 Β±0.08 / 12.30 ms β no change β
β QQuery 62 β 44.37 / 45.02 Β±0.60 / 46.14 ms β 43.96 / 45.84 Β±2.82 / 51.44 ms β no change β
β QQuery 63 β 29.80 / 30.13 Β±0.37 / 30.85 ms β 29.85 / 30.15 Β±0.18 / 30.35 ms β no change β
β QQuery 64 β 361.01 / 365.63 Β±4.05 / 371.38 ms β 360.95 / 365.38 Β±2.95 / 368.67 ms β no change β
β QQuery 65 β 132.37 / 132.88 Β±0.47 / 133.69 ms β 131.45 / 133.74 Β±1.66 / 135.73 ms β no change β
β QQuery 66 β 77.89 / 79.34 Β±1.61 / 82.11 ms β 77.38 / 79.50 Β±2.53 / 84.42 ms β no change β
β QQuery 67 β 294.00 / 312.01 Β±15.44 / 335.17 ms β 296.14 / 302.63 Β±5.24 / 310.53 ms β no change β
β QQuery 68 β 12.05 / 12.14 Β±0.08 / 12.24 ms β 12.19 / 12.68 Β±0.31 / 12.96 ms β no change β
β QQuery 69 β 56.64 / 57.48 Β±0.95 / 59.32 ms β 56.48 / 57.51 Β±0.71 / 58.64 ms β no change β
β QQuery 70 β 111.11 / 113.20 Β±2.61 / 118.21 ms β 109.22 / 111.25 Β±3.15 / 117.48 ms β no change β
β QQuery 71 β 34.98 / 35.46 Β±0.41 / 36.21 ms β 35.48 / 36.19 Β±0.55 / 36.88 ms β no change β
β QQuery 72 β 1696.97 / 1799.98 Β±62.74 / 1871.63 ms β 1716.93 / 1733.42 Β±15.81 / 1756.05 ms β no change β
β QQuery 73 β 9.76 / 10.20 Β±0.39 / 10.87 ms β 10.09 / 10.51 Β±0.24 / 10.78 ms β no change β
β QQuery 74 β 160.42 / 168.71 Β±4.69 / 173.03 ms β 165.08 / 168.79 Β±3.54 / 175.22 ms β no change β
β QQuery 75 β 142.08 / 142.87 Β±0.81 / 144.06 ms β 142.08 / 142.92 Β±0.92 / 144.69 ms β no change β
β QQuery 76 β 33.96 / 34.63 Β±0.58 / 35.29 ms β 35.14 / 36.02 Β±1.20 / 38.37 ms β no change β
β QQuery 77 β 61.57 / 63.13 Β±2.11 / 67.32 ms β 59.98 / 61.18 Β±0.66 / 61.93 ms β no change β
β QQuery 78 β 165.59 / 170.81 Β±3.57 / 175.55 ms β 169.00 / 172.50 Β±2.71 / 176.34 ms β no change β
β QQuery 79 β 67.53 / 68.31 Β±1.01 / 70.22 ms β 66.81 / 67.75 Β±0.82 / 69.27 ms β no change β
β QQuery 80 β 96.52 / 98.47 Β±1.13 / 99.71 ms β 95.61 / 98.39 Β±1.77 / 101.18 ms β no change β
β QQuery 81 β 25.89 / 26.57 Β±0.36 / 26.88 ms β 26.91 / 27.22 Β±0.29 / 27.63 ms β no change β
β QQuery 82 β 17.12 / 17.31 Β±0.17 / 17.62 ms β 16.95 / 17.21 Β±0.21 / 17.56 ms β no change β
β QQuery 83 β 33.53 / 34.91 Β±1.37 / 37.31 ms β 33.32 / 33.98 Β±0.65 / 34.90 ms β no change β
β QQuery 84 β 29.94 / 30.25 Β±0.22 / 30.49 ms β 29.54 / 30.43 Β±0.94 / 32.06 ms β no change β
β QQuery 85 β 102.46 / 103.42 Β±0.60 / 104.10 ms β 103.10 / 103.87 Β±0.70 / 104.97 ms β no change β
β QQuery 86 β 25.97 / 26.29 Β±0.22 / 26.58 ms β 25.51 / 25.87 Β±0.41 / 26.67 ms β no change β
β QQuery 87 β 61.61 / 64.87 Β±2.09 / 67.64 ms β 60.43 / 61.49 Β±0.71 / 62.56 ms β +1.05x faster β
β QQuery 88 β 62.30 / 63.08 Β±0.80 / 64.52 ms β 61.62 / 62.22 Β±0.41 / 62.84 ms β no change β
β QQuery 89 β 35.51 / 36.20 Β±0.39 / 36.66 ms β 34.88 / 35.46 Β±0.51 / 36.36 ms β no change β
β QQuery 90 β 17.15 / 17.30 Β±0.11 / 17.42 ms β 17.21 / 17.51 Β±0.15 / 17.63 ms β no change β
β QQuery 91 β 45.64 / 46.48 Β±1.18 / 48.77 ms β 44.49 / 44.84 Β±0.38 / 45.47 ms β no change β
β QQuery 92 β 30.41 / 30.86 Β±0.59 / 31.93 ms β 29.44 / 30.28 Β±0.48 / 30.87 ms β no change β
β QQuery 93 β 49.53 / 50.05 Β±0.51 / 50.80 ms β 49.29 / 50.78 Β±0.92 / 51.79 ms β no change β
β QQuery 94 β 38.63 / 40.18 Β±1.11 / 41.37 ms β 38.39 / 38.89 Β±0.35 / 39.45 ms β no change β
β QQuery 95 β 81.58 / 82.38 Β±0.76 / 83.57 ms β 80.01 / 80.89 Β±0.83 / 82.18 ms β no change β
β QQuery 96 β 24.16 / 24.38 Β±0.18 / 24.61 ms β 23.51 / 23.87 Β±0.31 / 24.35 ms β no change β
β QQuery 97 β 51.00 / 51.93 Β±0.91 / 53.68 ms β 49.94 / 51.88 Β±2.01 / 55.76 ms β no change β
β QQuery 98 β 43.48 / 45.81 Β±1.73 / 48.05 ms β 42.63 / 42.91 Β±0.34 / 43.55 ms β +1.07x faster β
β QQuery 99 β 66.89 / 67.16 Β±0.26 / 67.60 ms β 65.59 / 66.19 Β±0.44 / 66.87 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9464.03ms β
β Total Time (spill-dedup-view-arrays) β 9393.03ms β
β Average Time (HEAD) β 95.60ms β
β Average Time (spill-dedup-view-arrays) β 94.88ms β
β Queries Faster β 4 β
β Queries Slower β 2 β
β Queries with No Change β 93 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`tpcds` β `tpcds_sf1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 1 | 192.6 KiB | 192.6 KiB | +0.0% |
| Query 2 | 91.5 MiB | 99.5 MiB | +8.8% |
| Query 3 | 3.4 MiB | 3.4 MiB | +0.0% |
| Query 4 | 120.5 MiB | 120.5 MiB | +0.0% |
| Query 5 | 13.0 MiB | 13.0 MiB | +0.2% |
| Query 6 | 7.6 MiB | 7.6 MiB | -0.0% |
| Query 7 | 9.1 MiB | 8.3 MiB | -8.3% |
| Query 8 | 2.5 MiB | 2.4 MiB | -2.3% |
| Query 9 | 32.1 KiB | 32.1 KiB | +0.0% |
| Query 10 | 2.9 MiB | 2.9 MiB | +0.0% |
| Query 11 | 83.7 MiB | 83.7 MiB | +0.0% |
| Query 12 | 130.3 MiB | 131.1 MiB | +0.7% |
| Query 13 | 15.4 MiB | 16.1 MiB | +4.1% |
| Query 14 | 170.3 MiB | 171.4 MiB | +0.7% |
| Query 15 | 6.5 MiB | 6.7 MiB | +3.9% |
| Query 16 | 161.6 KiB | 161.6 KiB | +0.0% |
| Query 17 | 40.4 MiB | 42.8 MiB | +5.7% |
| Query 18 | 19.3 MiB | 19.7 MiB | +2.3% |
| Query 19 | 5.0 MiB | 5.0 MiB | -0.0% |
| Query 20 | 131.3 MiB | 131.3 MiB | +0.0% |
| Query 21 | 5.3 MiB | 5.3 MiB | +0.9% |
| Query 22 | 91.6 MiB | 92.3 MiB | +0.7% |
| Query 23 | 205.6 MiB | 204.3 MiB | -0.6% |
| Query 24 | 48.1 MiB | 48.1 MiB | +0.0% |
| Query 25 | 99.1 MiB | 99.1 MiB | +0.0% |
| Query 26 | 9.7 MiB | 9.7 MiB | -0.0% |
| Query 27 | 1.7 MiB | 1.7 MiB | +0.0% |
| Query 28 | 1.3 MiB | 1.2 MiB | -9.9% |
| Query 29 | 104.4 MiB | 104.4 MiB | +0.0% |
| Query 30 | 6.7 MiB | 6.7 MiB | -0.0% |
| Query 31 | 17.8 MiB | 17.8 MiB | +0.0% |
| Query 32 | 2.6 MiB | 2.6 MiB | +0.0% |
| Query 33 | 4.1 MiB | 3.9 MiB | -6.7% |
| Query 34 | 14.3 MiB | 14.3 MiB | +0.0% |
| Query 35 | 25.3 MiB | 25.3 MiB | +0.0% |
| Query 36 | 10.6 MiB | 10.6 MiB | +0.0% |
| Query 37 | 3.8 MiB | 3.8 MiB | +0.0% |
| Query 38 | 20.1 MiB | 20.1 MiB | +0.0% |
| Query 39 | 42.9 MiB | 73.3 MiB | +71.0% |
| Query 40 | 13.5 MiB | 13.5 MiB | +0.0% |
| Query 41 | 2.0 MiB | 1.9 MiB | -7.7% |
| Query 42 | 1.6 MiB | 2.0 MiB | +19.0% |
| Query 43 | 192.4 KiB | 192.4 KiB | +0.0% |
| Query 44 | 192.4 KiB | 192.4 KiB | +0.0% |
| Query 45 | 8.1 MiB | 8.5 MiB | +4.7% |
| Query 46 | 10.1 MiB | 10.1 MiB | +0.0% |
| Query 47 | 155.9 MiB | 146.8 MiB | -5.9% |
| Query 48 | 16.2 MiB | 15.8 MiB | -2.9% |
| Query 49 | 52.0 MiB | 51.6 MiB | -0.8% |
| Query 50 | 15.5 MiB | 15.2 MiB | -2.3% |
| Query 51 | 178.6 MiB | 178.6 MiB | -0.0% |
| Query 52 | 3.3 MiB | 3.3 MiB | +0.0% |
| Query 53 | 30.6 MiB | 30.6 MiB | +0.0% |
| Query 54 | 4.3 MiB | 4.2 MiB | -3.6% |
| Query 55 | 2.9 MiB | 3.2 MiB | +9.8% |
| Query 56 | 4.0 MiB | 4.0 MiB | +0.9% |
| Query 57 | 141.9 MiB | 141.9 MiB | +0.0% |
| Query 58 | 6.2 MiB | 5.9 MiB | -4.2% |
| Query 59 | 15.0 MiB | 15.0 MiB | +0.0% |
| Query 60 | 5.6 MiB | 5.9 MiB | +3.8% |
| Query 61 | 1.0 MiB | 880.8 KiB | -17.9% |
| Query 62 | 7.0 MiB | 7.7 MiB | +9.9% |
| Query 63 | 20.6 MiB | 20.6 MiB | -0.0% |
| Query 64 | 100.1 MiB | 107.8 MiB | +7.7% |
| Query 65 | 28.3 MiB | 28.3 MiB | -0.0% |
| Query 66 | 12.2 MiB | 15.9 MiB | +30.5% |
| Query 67 | 304.8 MiB | 362.0 MiB | +18.8% |
| Query 68 | 10.1 MiB | 10.1 MiB | +0.0% |
| Query 69 | 46.6 MiB | 46.6 MiB | -0.0% |
| Query 70 | 72.5 MiB | 68.4 MiB | -5.7% |
| Query 71 | 40.4 MiB | 60.4 MiB | +49.5% |
| Query 72 | 96.9 MiB | 96.9 MiB | +0.0% |
| Query 73 | 14.3 MiB | 14.3 MiB | +0.0% |
| Query 74 | 41.6 MiB | 41.6 MiB | +0.0% |
| Query 75 | 20.3 MiB | 20.3 MiB | +0.0% |
| Query 76 | 7.7 MiB | 7.6 MiB | -0.3% |
| Query 77 | 3.4 MiB | 3.4 MiB | -0.0% |
| Query 78 | 109.9 MiB | 110.7 MiB | +0.8% |
| Query 79 | 11.6 MiB | 10.8 MiB | -6.8% |
| Query 80 | 40.0 MiB | 39.6 MiB | -1.0% |
| Query 81 | 10.7 MiB | 10.7 MiB | +0.0% |
| Query 82 | 3.8 MiB | 3.8 MiB | +0.0% |
| Query 83 | 11.2 MiB | 11.2 MiB | +0.0% |
| Query 84 | 11.9 MiB | 11.9 MiB | +0.0% |
| Query 85 | 19.3 MiB | 19.6 MiB | +1.5% |
| Query 86 | 105.2 MiB | 105.2 MiB | +0.0% |
| Query 87 | 20.3 MiB | 20.3 MiB | +0.0% |
| Query 88 | 1.5 MiB | 1.7 MiB | +11.1% |
| Query 89 | 130.0 MiB | 130.0 MiB | +0.0% |
| Query 90 | 562.1 KiB | 562.1 KiB | +0.0% |
| Query 91 | 22.7 MiB | 22.7 MiB | +0.0% |
| Query 92 | 2.9 MiB | 2.6 MiB | -7.7% |
| Query 93 | 15.4 MiB | 15.8 MiB | +2.4% |
| Query 94 | 3.4 MiB | 3.7 MiB | +8.4% |
| Query 95 | 18.3 MiB | 17.1 MiB | -6.1% |
| Query 96 | 224.7 KiB | 224.7 KiB | +0.0% |
| Query 97 | 32.3 MiB | 32.3 MiB | +0.0% |
| Query 98 | 131.5 MiB | 131.5 MiB | +0.0% |
| Query 99 | 7.8 MiB | 7.1 MiB | -8.9% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| tpcds | base (`ed43e69 (merge-base)`) | 304.8 MiB | 3.1 GiB | 2.8 GiB | 10.4Γ |
| tpcds | changed (`spill-dedup-view-arrays`) | 362.0 MiB | 3.6 GiB | 3.2 GiB | 10.1Γ |
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 3.1 GiB |
| Avg memory | 2.1 GiB |
| CPU user | 207.7s |
| CPU sys | 6.7s |
| Peak spill | 86.9 MiB |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 3.6 GiB |
| Avg memory | 2.1 GiB |
| CPU user | 207.5s |
| CPU sys | 6.7s |
| Peak spill | 55.1 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310
run benchmarks
```yaml
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783142668
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783076971-2589-2vchs 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783741023
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783724486-2598-qxmn2 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784537096
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784306558)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q04_sort_string_all_distinct_distinct 1.00 137.7Β±0.65ms ? ?/sec 1.25 172.5Β±12.53ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 350.2s |
| Peak memory | 420.5 MiB |
| Avg memory | 14.3 MiB |
| CPU user | 11.4s |
| CPU sys | 6.6s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 340.2s |
| Peak memory | 374.8 MiB |
| Avg memory | 15.1 MiB |
| CPU user | 11.1s |
| CPU sys | 5.0s |
| Peak spill | 74.0 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783728693
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783730184
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783900146
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.90 ms β 5.90 ms β no change β
β QQuery 2 β 81.92 ms β 81.50 ms β no change β
β QQuery 3 β 29.50 ms β 29.15 ms β no change β
β QQuery 4 β 476.78 ms β 436.56 ms β +1.09x faster β
β QQuery 5 β 51.72 ms β 50.67 ms β no change β
β QQuery 6 β 35.74 ms β 34.79 ms β no change β
β QQuery 7 β 74.66 ms β 73.74 ms β no change β
β QQuery 8 β 37.39 ms β 35.96 ms β no change β
β QQuery 9 β 51.67 ms β 52.31 ms β no change β
β QQuery 10 β 62.39 ms β 61.58 ms β no change β
β QQuery 11 β 306.52 ms β 267.70 ms β +1.15x faster β
β QQuery 12 β 29.34 ms β 28.63 ms β no change β
β QQuery 13 β 118.88 ms β 117.69 ms β no change β
β QQuery 14 β 399.27 ms β 392.36 ms β no change β
β QQuery 15 β 56.90 ms β 51.97 ms β +1.09x faster β
β QQuery 16 β 6.82 ms β 6.40 ms β +1.07x faster β
β QQuery 17 β 78.58 ms β 78.37 ms β no change β
β QQuery 18 β 104.89 ms β 102.06 ms β no change β
β QQuery 19 β 41.37 ms β 40.59 ms β no change β
β QQuery 20 β 36.12 ms β 35.83 ms β no change β
β QQuery 21 β 17.58 ms β 17.39 ms β no change β
β QQuery 22 β 65.02 ms β 63.65 ms β no change β
β QQuery 23 β 310.49 ms β 308.80 ms β no change β
β QQuery 24 β 202.83 ms β 204.89 ms β no change β
β QQuery 25 β 108.32 ms β 108.86 ms β no change β
β QQuery 26 β 47.63 ms β 49.09 ms β no change β
β QQuery 27 β 6.40 ms β 6.47 ms β no change β
β QQuery 28 β 57.18 ms β 57.63 ms β no change β
β QQuery 29 β 93.97 ms β 97.59 ms β no change β
β QQuery 30 β 32.20 ms β 33.84 ms β 1.05x slower β
β QQuery 31 β 109.07 ms β 112.33 ms β no change β
β QQuery 32 β 21.13 ms β 21.84 ms β no change β
β QQuery 33 β 37.94 ms β 38.60 ms β no change β
β QQuery 34 β 10.10 ms β 10.44 ms β no change β
β QQuery 35 β 71.34 ms β 75.56 ms β 1.06x slower β
β QQuery 36 β 5.72 ms β 6.04 ms β 1.06x slower β
β QQuery 37 β 6.81 ms β 7.25 ms β 1.06x slower β
β QQuery 38 β 61.23 ms β 64.34 ms β 1.05x slower β
β QQuery 39 β 76.04 ms β 76.02 ms β no change β
β QQuery 40 β 24.17 ms β 24.42 ms β no change β
β QQuery 41 β 11.58 ms β 11.63 ms β no change β
β QQuery 42 β 24.41 ms β 24.39 ms β no change β
β QQuery 43 β 5.42 ms β 5.42 ms β no change β
β QQuery 44 β 9.67 ms β 9.71 ms β no change β
β QQuery 45 β 41.53 ms β 41.10 ms β no change β
β QQuery 46 β 12.36 ms β 12.31 ms β no change β
β QQuery 47 β 232.50 ms β 238.94 ms β no change β
β QQuery 48 β 95.55 ms β 96.07 ms β no change β
β QQuery 49 β 73.18 ms β 71.25 ms β no change β
β QQuery 50 β 59.86 ms β 58.19 ms β no change β
β QQuery 51 β 95.05 ms β 91.85 ms β no change β
β QQuery 52 β 24.81 ms β 25.37 ms β no change β
β QQuery 53 β 29.63 ms β 29.86 ms β no change β
β QQuery 54 β 55.09 ms β 55.68 ms β no change β
β QQuery 55 β 24.16 ms β 24.55 ms β no change β
β QQuery 56 β 39.50 ms β 39.89 ms β no change β
β QQuery 57 β 172.35 ms β 174.66 ms β no change β
β QQuery 58 β 111.45 ms β 110.03 ms β no change β
β QQuery 59 β 118.29 ms β 117.86 ms β no change β
β QQuery 60 β 40.33 ms β 39.82 ms β no change β
β QQuery 61 β 11.89 ms β 12.06 ms β no change β
β QQuery 62 β 45.34 ms β 45.68 ms β no change β
β QQuery 63 β 29.31 ms β 29.44 ms β no change β
β QQuery 64 β 358.35 ms β 369.77 ms β no change β
β QQuery 65 β 130.25 ms β 134.52 ms β no change β
β QQuery 66 β 77.92 ms β 79.50 ms β no change β
β QQuery 67 β 256.34 ms β 261.55 ms β no change β
β QQuery 68 β 11.70 ms β 12.19 ms β no change β
β QQuery 69 β 57.63 ms β 56.49 ms β no change β
β QQuery 70 β 104.84 ms β 105.11 ms β no change β
β QQuery 71 β 36.02 ms β 36.29 ms β no change β
β QQuery 72 β 1800.11 ms β 1753.48 ms β no change β
β QQuery 73 β 10.40 ms β 9.55 ms β +1.09x faster β
β QQuery 74 β 172.74 ms β 153.86 ms β +1.12x faster β
β QQuery 75 β 142.24 ms β 141.80 ms β no change β
β QQuery 76 β 35.19 ms β 34.59 ms β no change β
β QQuery 77 β 60.74 ms β 59.79 ms β no change β
β QQuery 78 β 160.61 ms β 161.27 ms β no change β
β QQuery 79 β 66.20 ms β 65.55 ms β no change β
β QQuery 80 β 97.65 ms β 94.74 ms β no change β
β QQuery 81 β 25.56 ms β 25.78 ms β no change β
β QQuery 82 β 16.36 ms β 17.20 ms β 1.05x slower β
β QQuery 83 β 33.42 ms β 34.66 ms β no change β
β QQuery 84 β 29.62 ms β 30.25 ms β no change β
β QQuery 85 β 105.10 ms β 105.23 ms β no change β
β QQuery 86 β 26.00 ms β 25.96 ms β no change β
β QQuery 87 β 65.02 ms β 63.88 ms β no change β
β QQuery 88 β 62.15 ms β 61.90 ms β no change β
β QQuery 89 β 35.98 ms β 35.68 ms β no change β
β QQuery 90 β 17.20 ms β 17.33 ms β no change β
β QQuery 91 β 44.96 ms β 44.67 ms β no change β
β QQuery 92 β 29.90 ms β 30.48 ms β no change β
β QQuery 93 β 50.65 ms β 50.35 ms β no change β
β QQuery 94 β 39.62 ms β 39.75 ms β no change β
β QQuery 95 β 81.00 ms β 81.59 ms β no change β
β QQuery 96 β 24.13 ms β 24.08 ms β no change β
β QQuery 97 β 51.53 ms β 51.62 ms β no change β
β QQuery 98 β 44.07 ms β 43.86 ms β no change β
β QQuery 99 β 67.33 ms β 67.55 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9243.36ms β
β Total Time (spill-dedup-view-arrays) β 9116.40ms β
β Average Time (HEAD) β 93.37ms β
β Average Time (spill-dedup-view-arrays) β 92.08ms β
β Queries Faster β 6 β
β Queries Slower β 6 β
β Queries with No Change β 87 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.90 / 6.45 Β±0.94 / 8.32 ms β 5.90 / 6.40 Β±0.88 / 8.15 ms β no change β
β QQuery 2 β 81.92 / 82.34 Β±0.38 / 82.81 ms β 81.50 / 82.17 Β±0.53 / 82.91 ms β no change β
β QQuery 3 β 29.50 / 29.73 Β±0.19 / 29.92 ms β 29.15 / 29.33 Β±0.22 / 29.74 ms β no change β
β QQuery 4 β 476.78 / 479.87 Β±2.45 / 483.43 ms β 436.56 / 447.58 Β±8.07 / 455.85 ms β +1.07x faster β
β QQuery 5 β 51.72 / 54.16 Β±3.89 / 61.89 ms β 50.67 / 51.17 Β±0.25 / 51.37 ms β +1.06x faster β
β QQuery 6 β 35.74 / 36.07 Β±0.19 / 36.27 ms β 34.79 / 35.44 Β±0.37 / 35.88 ms β no change β
β QQuery 7 β 74.66 / 75.69 Β±0.58 / 76.32 ms β 73.74 / 74.44 Β±0.60 / 75.24 ms β no change β
β QQuery 8 β 37.39 / 38.76 Β±2.20 / 43.13 ms β 35.96 / 36.15 Β±0.16 / 36.37 ms β +1.07x faster β
β QQuery 9 β 51.67 / 53.65 Β±1.57 / 55.47 ms β 52.31 / 53.09 Β±0.49 / 53.79 ms β no change β
β QQuery 10 β 62.39 / 63.02 Β±0.70 / 64.35 ms β 61.58 / 62.05 Β±0.26 / 62.35 ms β no change β
β QQuery 11 β 306.52 / 316.48 Β±7.87 / 324.89 ms β 267.70 / 273.53 Β±8.98 / 291.40 ms β +1.16x faster β
β QQuery 12 β 29.34 / 29.56 Β±0.15 / 29.81 ms β 28.63 / 29.09 Β±0.44 / 29.62 ms β no change β
β QQuery 13 β 118.88 / 119.26 Β±0.28 / 119.61 ms β 117.69 / 118.93 Β±0.67 / 119.44 ms β no change β
β QQuery 14 β 399.27 / 404.85 Β±3.22 / 408.88 ms β 392.36 / 397.88 Β±3.75 / 402.88 ms β no change β
β QQuery 15 β 56.90 / 58.61 Β±1.38 / 61.01 ms β 51.97 / 53.46 Β±1.05 / 54.98 ms β +1.10x faster β
β QQuery 16 β 6.82 / 6.93 Β±0.13 / 7.18 ms β 6.40 / 6.58 Β±0.21 / 7.00 ms β +1.05x faster β
β QQuery 17 β 78.58 / 80.82 Β±2.42 / 85.23 ms β 78.37 / 79.32 Β±0.67 / 80.43 ms β no change β
β QQuery 18 β 104.89 / 106.79 Β±1.47 / 108.71 ms β 102.06 / 104.39 Β±2.26 / 108.72 ms β no change β
β QQuery 19 β 41.37 / 41.99 Β±0.63 / 43.12 ms β 40.59 / 41.41 Β±0.47 / 42.03 ms β no change β
β QQuery 20 β 36.12 / 36.65 Β±0.37 / 37.26 ms β 35.83 / 36.29 Β±0.43 / 37.10 ms β no change β
β QQuery 21 β 17.58 / 18.76 Β±1.94 / 22.62 ms β 17.39 / 17.52 Β±0.08 / 17.64 ms β +1.07x faster β
β QQuery 22 β 65.02 / 66.26 Β±1.36 / 68.66 ms β 63.65 / 66.60 Β±1.65 / 68.44 ms β no change β
β QQuery 23 β 310.49 / 324.55 Β±9.28 / 332.22 ms β 308.80 / 319.40 Β±9.56 / 333.95 ms β no change β
β QQuery 24 β 202.83 / 207.13 Β±3.36 / 211.29 ms β 204.89 / 210.17 Β±7.18 / 224.36 ms β no change β
β QQuery 25 β 108.32 / 109.87 Β±1.90 / 113.44 ms β 108.86 / 111.26 Β±2.26 / 114.56 ms β no change β
β QQuery 26 β 47.63 / 48.19 Β±0.42 / 48.78 ms β 49.09 / 49.36 Β±0.30 / 49.86 ms β no change β
β QQuery 27 β 6.40 / 6.52 Β±0.11 / 6.71 ms β 6.47 / 6.65 Β±0.11 / 6.77 ms β no change β
β QQuery 28 β 57.18 / 61.42 Β±2.15 / 63.10 ms β 57.63 / 61.36 Β±1.88 / 62.72 ms β no change β
β QQuery 29 β 93.97 / 96.43 Β±2.05 / 100.03 ms β 97.59 / 100.10 Β±2.74 / 105.02 ms β no change β
β QQuery 30 β 32.20 / 32.36 Β±0.18 / 32.71 ms β 33.84 / 34.54 Β±0.70 / 35.67 ms β 1.07x slower β
β QQuery 31 β 109.07 / 112.43 Β±2.77 / 117.49 ms β 112.33 / 112.95 Β±0.56 / 113.98 ms β no change β
β QQuery 32 β 21.13 / 21.77 Β±1.00 / 23.76 ms β 21.84 / 24.02 Β±3.63 / 31.27 ms β 1.10x slower β
β QQuery 33 β 37.94 / 38.12 Β±0.13 / 38.36 ms β 38.60 / 39.56 Β±0.98 / 41.30 ms β no change β
β QQuery 34 β 10.10 / 10.33 Β±0.14 / 10.50 ms β 10.44 / 10.88 Β±0.32 / 11.42 ms β 1.05x slower β
β QQuery 35 β 71.34 / 73.43 Β±2.80 / 78.94 ms β 75.56 / 76.12 Β±0.56 / 76.96 ms β no change β
β QQuery 36 β 5.72 / 5.86 Β±0.20 / 6.25 ms β 6.04 / 6.13 Β±0.10 / 6.32 ms β no change β
β QQuery 37 β 6.81 / 6.92 Β±0.09 / 7.08 ms β 7.25 / 7.35 Β±0.06 / 7.44 ms β 1.06x slower β
β QQuery 38 β 61.23 / 62.52 Β±0.68 / 63.18 ms β 64.34 / 65.87 Β±1.05 / 67.25 ms β 1.05x slower β
β QQuery 39 β 76.04 / 76.55 Β±0.37 / 77.03 ms β 76.02 / 76.82 Β±0.74 / 77.86 ms β no change β
β QQuery 40 β 24.17 / 24.83 Β±0.64 / 26.01 ms β 24.42 / 24.92 Β±0.33 / 25.45 ms β no change β
β QQuery 41 β 11.58 / 11.74 Β±0.11 / 11.89 ms β 11.63 / 11.76 Β±0.08 / 11.86 ms β no change β
β QQuery 42 β 24.41 / 26.22 Β±2.35 / 30.82 ms β 24.39 / 24.93 Β±0.68 / 26.18 ms β no change β
β QQuery 43 β 5.42 / 5.54 Β±0.09 / 5.68 ms β 5.42 / 5.49 Β±0.06 / 5.60 ms β no change β
β QQuery 44 β 9.67 / 9.70 Β±0.03 / 9.75 ms β 9.71 / 9.75 Β±0.02 / 9.78 ms β no change β
β QQuery 45 β 41.53 / 42.52 Β±0.69 / 43.39 ms β 41.10 / 43.42 Β±1.96 / 46.59 ms β no change β
β QQuery 46 β 12.36 / 12.66 Β±0.26 / 13.02 ms β 12.31 / 12.79 Β±0.50 / 13.63 ms β no change β
β QQuery 47 β 232.50 / 237.50 Β±3.99 / 243.75 ms β 238.94 / 241.80 Β±2.93 / 247.34 ms β no change β
β QQuery 48 β 95.55 / 97.72 Β±2.12 / 101.71 ms β 96.07 / 96.61 Β±0.39 / 97.17 ms β no change β
β QQuery 49 β 73.18 / 74.18 Β±1.06 / 76.12 ms β 71.25 / 74.48 Β±5.02 / 84.49 ms β no change β
β QQuery 50 β 59.86 / 60.40 Β±0.60 / 61.56 ms β 58.19 / 59.28 Β±1.43 / 62.03 ms β no change β
β QQuery 51 β 95.05 / 97.18 Β±1.78 / 99.78 ms β 91.85 / 93.81 Β±1.46 / 96.13 ms β no change β
β QQuery 52 β 24.81 / 25.80 Β±0.95 / 27.48 ms β 25.37 / 28.14 Β±3.92 / 35.90 ms β 1.09x slower β
β QQuery 53 β 29.63 / 29.82 Β±0.13 / 29.98 ms β 29.86 / 29.96 Β±0.08 / 30.09 ms β no change β
β QQuery 54 β 55.09 / 55.40 Β±0.33 / 55.98 ms β 55.68 / 56.14 Β±0.42 / 56.66 ms β no change β
β QQuery 55 β 24.16 / 24.31 Β±0.14 / 24.57 ms β 24.55 / 24.75 Β±0.26 / 25.27 ms β no change β
β QQuery 56 β 39.50 / 40.00 Β±0.32 / 40.44 ms β 39.89 / 40.54 Β±0.54 / 41.26 ms β no change β
β QQuery 57 β 172.35 / 175.58 Β±3.10 / 181.19 ms β 174.66 / 175.70 Β±0.73 / 176.49 ms β no change β
β QQuery 58 β 111.45 / 112.70 Β±0.94 / 114.34 ms β 110.03 / 113.07 Β±2.76 / 118.06 ms β no change β
β QQuery 59 β 118.29 / 120.53 Β±2.56 / 125.31 ms β 117.86 / 118.44 Β±0.55 / 119.35 ms β no change β
β QQuery 60 β 40.33 / 41.38 Β±1.08 / 43.19 ms β 39.82 / 40.99 Β±0.64 / 41.54 ms β no change β
β QQuery 61 β 11.89 / 12.06 Β±0.16 / 12.34 ms β 12.06 / 12.28 Β±0.14 / 12.46 ms β no change β
β QQuery 62 β 45.34 / 45.47 Β±0.13 / 45.71 ms β 45.68 / 46.07 Β±0.30 / 46.55 ms β no change β
β QQuery 63 β 29.31 / 29.56 Β±0.18 / 29.81 ms β 29.44 / 29.94 Β±0.28 / 30.29 ms β no change β
β QQuery 64 β 358.35 / 371.64 Β±9.50 / 384.26 ms β 369.77 / 372.12 Β±2.36 / 376.59 ms β no change β
β QQuery 65 β 130.25 / 139.18 Β±5.47 / 146.91 ms β 134.52 / 136.24 Β±1.23 / 138.17 ms β no change β
β QQuery 66 β 77.92 / 78.41 Β±0.51 / 79.22 ms β 79.50 / 79.64 Β±0.13 / 79.87 ms β no change β
β QQuery 67 β 256.34 / 259.00 Β±1.85 / 262.02 ms β 261.55 / 270.42 Β±7.56 / 284.25 ms β no change β
β QQuery 68 β 11.70 / 12.04 Β±0.25 / 12.44 ms β 12.19 / 12.36 Β±0.11 / 12.54 ms β no change β
β QQuery 69 β 57.63 / 59.19 Β±1.70 / 62.19 ms β 56.49 / 59.34 Β±4.10 / 67.43 ms β no change β
β QQuery 70 β 104.84 / 105.71 Β±0.61 / 106.77 ms β 105.11 / 107.52 Β±2.00 / 111.19 ms β no change β
β QQuery 71 β 36.02 / 37.07 Β±1.01 / 38.77 ms β 36.29 / 36.79 Β±0.38 / 37.46 ms β no change β
β QQuery 72 β 1800.11 / 1910.25 Β±81.52 / 2028.78 ms β 1753.48 / 1831.85 Β±43.85 / 1887.90 ms β no change β
β QQuery 73 β 10.40 / 10.71 Β±0.23 / 11.07 ms β 9.55 / 10.58 Β±0.71 / 11.58 ms β no change β
β QQuery 74 β 172.74 / 176.16 Β±1.92 / 178.44 ms β 153.86 / 156.85 Β±2.17 / 160.04 ms β +1.12x faster β
β QQuery 75 β 142.24 / 145.17 Β±3.34 / 151.27 ms β 141.80 / 147.49 Β±5.29 / 157.34 ms β no change β
β QQuery 76 β 35.19 / 35.91 Β±0.92 / 37.69 ms β 34.59 / 34.85 Β±0.26 / 35.32 ms β no change β
β QQuery 77 β 60.74 / 61.71 Β±0.74 / 62.87 ms β 59.79 / 63.18 Β±5.90 / 74.95 ms β no change β
β QQuery 78 β 160.61 / 169.32 Β±9.62 / 187.94 ms β 161.27 / 164.98 Β±3.84 / 171.88 ms β no change β
β QQuery 79 β 66.20 / 66.51 Β±0.26 / 66.84 ms β 65.55 / 67.87 Β±3.20 / 74.16 ms β no change β
β QQuery 80 β 97.65 / 101.16 Β±6.76 / 114.67 ms β 94.74 / 95.79 Β±1.28 / 98.27 ms β +1.06x faster β
β QQuery 81 β 25.56 / 25.87 Β±0.20 / 26.16 ms β 25.78 / 26.38 Β±0.33 / 26.76 ms β no change β
β QQuery 82 β 16.36 / 16.54 Β±0.17 / 16.80 ms β 17.20 / 17.68 Β±0.85 / 19.36 ms β 1.07x slower β
β QQuery 83 β 33.42 / 33.80 Β±0.28 / 34.19 ms β 34.66 / 37.10 Β±4.36 / 45.82 ms β 1.10x slower β
β QQuery 84 β 29.62 / 32.26 Β±4.68 / 41.61 ms β 30.25 / 30.76 Β±0.65 / 32.03 ms β no change β
β QQuery 85 β 105.10 / 107.35 Β±1.62 / 109.41 ms β 105.23 / 106.21 Β±0.68 / 107.04 ms β no change β
β QQuery 86 β 26.00 / 26.38 Β±0.30 / 26.88 ms β 25.96 / 26.28 Β±0.35 / 26.94 ms β no change β
β QQuery 87 β 65.02 / 65.44 Β±0.25 / 65.78 ms β 63.88 / 66.39 Β±2.60 / 71.21 ms β no change β
β QQuery 88 β 62.15 / 63.63 Β±2.39 / 68.39 ms β 61.90 / 62.70 Β±0.87 / 64.35 ms β no change β
β QQuery 89 β 35.98 / 36.87 Β±0.82 / 38.37 ms β 35.68 / 36.25 Β±0.37 / 36.75 ms β no change β
β QQuery 90 β 17.20 / 17.26 Β±0.07 / 17.37 ms β 17.33 / 17.42 Β±0.11 / 17.63 ms β no change β
β QQuery 91 β 44.96 / 45.23 Β±0.23 / 45.56 ms β 44.67 / 46.40 Β±2.81 / 52.00 ms β no change β
β QQuery 92 β 29.90 / 30.78 Β±0.57 / 31.29 ms β 30.48 / 31.59 Β±0.83 / 32.62 ms β no change β
β QQuery 93 β 50.65 / 53.76 Β±4.24 / 62.03 ms β 50.35 / 50.92 Β±0.77 / 52.44 ms β +1.06x faster β
β QQuery 94 β 39.62 / 40.15 Β±0.50 / 40.95 ms β 39.75 / 40.21 Β±0.38 / 40.76 ms β no change β
β QQuery 95 β 81.00 / 82.06 Β±0.79 / 83.41 ms β 81.59 / 84.17 Β±3.92 / 91.97 ms β no change β
β QQuery 96 β 24.13 / 24.23 Β±0.10 / 24.42 ms β 24.08 / 24.42 Β±0.30 / 24.95 ms β no change β
β QQuery 97 β 51.53 / 53.87 Β±1.87 / 56.32 ms β 51.62 / 53.12 Β±1.09 / 54.43 ms β no change β
β QQuery 98 β 44.07 / 44.58 Β±0.50 / 45.37 ms β 43.86 / 44.87 Β±0.58 / 45.51 ms β no change β
β QQuery 99 β 67.33 / 67.80 Β±0.28 / 68.09 ms β 67.55 / 70.30 Β±4.49 / 79.24 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 9520.93ms β
β Total Time (spill-dedup-view-arrays) β 9351.44ms β
β Average Time (HEAD) β 96.17ms β
β Average Time (spill-dedup-view-arrays) β 94.46ms β
β Queries Faster β 10 β
β Queries Slower β 8 β
β Queries with No Change β 81 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.1 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 202.8s |
| CPU sys | 6.1s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.1 GiB |
| Avg memory | 1.4 GiB |
| CPU user | 200.5s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783898384
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 40.83 ms β 40.45 ms β no change β
β QQuery 2 β 18.44 ms β 18.52 ms β no change β
β QQuery 3 β 28.85 ms β 28.27 ms β no change β
β QQuery 4 β 17.34 ms β 17.35 ms β no change β
β QQuery 5 β 34.88 ms β 35.24 ms β no change β
β QQuery 6 β 16.13 ms β 16.22 ms β no change β
β QQuery 7 β 41.20 ms β 41.13 ms β no change β
β QQuery 8 β 39.88 ms β 39.97 ms β no change β
β QQuery 9 β 48.15 ms β 48.96 ms β no change β
β QQuery 10 β 40.35 ms β 40.53 ms β no change β
β QQuery 11 β 13.46 ms β 13.49 ms β no change β
β QQuery 12 β 20.99 ms β 20.67 ms β no change β
β QQuery 13 β 39.21 ms β 39.39 ms β no change β
β QQuery 14 β 24.47 ms β 24.56 ms β no change β
β QQuery 15 β 30.63 ms β 30.73 ms β no change β
β QQuery 16 β 13.87 ms β 13.79 ms β no change β
β QQuery 17 β 72.65 ms β 72.18 ms β no change β
β QQuery 18 β 61.28 ms β 59.88 ms β no change β
β QQuery 19 β 31.60 ms β 31.75 ms β no change β
β QQuery 20 β 31.65 ms β 31.85 ms β no change β
β QQuery 21 β 55.16 ms β 55.07 ms β no change β
β QQuery 22 β 13.80 ms β 13.99 ms β no change β
βββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 734.82ms β
β Total Time (spill-dedup-view-arrays) β 733.99ms β
β Average Time (HEAD) β 33.40ms β
β Average Time (spill-dedup-view-arrays) β 33.36ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 22 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 40.83 / 42.09 Β±1.62 / 45.17 ms β 40.45 / 41.62 Β±1.25 / 43.25 ms β no change β
β QQuery 2 β 18.44 / 18.79 Β±0.36 / 19.49 ms β 18.52 / 18.81 Β±0.27 / 19.28 ms β no change β
β QQuery 3 β 28.85 / 29.76 Β±0.96 / 31.61 ms β 28.27 / 28.68 Β±0.38 / 29.21 ms β no change β
β QQuery 4 β 17.34 / 17.72 Β±0.60 / 18.91 ms β 17.35 / 17.51 Β±0.23 / 17.94 ms β no change β
β QQuery 5 β 34.88 / 35.95 Β±1.05 / 37.89 ms β 35.24 / 35.85 Β±0.80 / 37.40 ms β no change β
β QQuery 6 β 16.13 / 16.30 Β±0.11 / 16.42 ms β 16.22 / 16.40 Β±0.19 / 16.74 ms β no change β
β QQuery 7 β 41.20 / 43.19 Β±1.85 / 46.18 ms β 41.13 / 42.70 Β±1.59 / 45.76 ms β no change β
β QQuery 8 β 39.88 / 40.49 Β±0.39 / 41.09 ms β 39.97 / 40.48 Β±0.35 / 40.89 ms β no change β
β QQuery 9 β 48.15 / 50.02 Β±1.27 / 51.89 ms β 48.96 / 49.80 Β±0.74 / 51.16 ms β no change β
β QQuery 10 β 40.35 / 40.50 Β±0.12 / 40.65 ms β 40.53 / 40.61 Β±0.08 / 40.76 ms β no change β
β QQuery 11 β 13.46 / 13.60 Β±0.15 / 13.88 ms β 13.49 / 13.72 Β±0.21 / 14.10 ms β no change β
β QQuery 12 β 20.99 / 21.33 Β±0.22 / 21.63 ms β 20.67 / 21.40 Β±0.61 / 22.28 ms β no change β
β QQuery 13 β 39.21 / 40.10 Β±0.57 / 40.93 ms β 39.39 / 40.78 Β±1.48 / 43.02 ms β no change β
β QQuery 14 β 24.47 / 24.57 Β±0.10 / 24.76 ms β 24.56 / 24.70 Β±0.13 / 24.94 ms β no change β
β QQuery 15 β 30.63 / 31.73 Β±1.02 / 33.40 ms β 30.73 / 31.53 Β±0.83 / 32.69 ms β no change β
β QQuery 16 β 13.87 / 14.14 Β±0.17 / 14.37 ms β 13.79 / 14.18 Β±0.36 / 14.80 ms β no change β
β QQuery 17 β 72.65 / 73.01 Β±0.35 / 73.51 ms β 72.18 / 73.03 Β±0.57 / 73.61 ms β no change β
β QQuery 18 β 61.28 / 62.28 Β±0.83 / 63.62 ms β 59.88 / 61.73 Β±1.56 / 63.73 ms β no change β
β QQuery 19 β 31.60 / 32.11 Β±0.69 / 33.46 ms β 31.75 / 32.01 Β±0.38 / 32.77 ms β no change β
β QQuery 20 β 31.65 / 32.15 Β±0.38 / 32.79 ms β 31.85 / 32.27 Β±0.39 / 32.98 ms β no change β
β QQuery 21 β 55.16 / 55.96 Β±1.25 / 58.43 ms β 55.07 / 55.99 Β±0.89 / 57.69 ms β no change β
β QQuery 22 β 13.80 / 14.13 Β±0.19 / 14.38 ms β 13.99 / 14.16 Β±0.12 / 14.33 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 749.93ms β
β Total Time (spill-dedup-view-arrays) β 747.95ms β
β Average Time (HEAD) β 34.09ms β
β Average Time (spill-dedup-view-arrays) β 34.00ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 22 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 518.6 MiB |
| CPU user | 21.0s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 517.3 MiB |
| CPU user | 20.9s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789948191
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789851638)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 7570366fd929daf9ced744bb8397686b50565b18 (7570366fd929daf9ced744bb8397686b50565b18) to 7570366fd929daf9ced744bb8397686b50565b18 [diff](https://github.com/apache/datafusion/compare/7570366fd929daf9ced744bb8397686b50565b18..7570366fd929daf9ced744bb8397686b50565b18)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
changed:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.88 ms β 134.90 ms β no change β
β Q2 β 139.37 ms β 137.61 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.65 ms β 200.15 ms β no change β
β Q5 β 233.11 ms β 228.56 ms β no change β
β Q6 β 256.76 ms β 252.52 ms β no change β
β Q7 β 695.03 ms β 673.63 ms β no change β
β Q8 β 478.25 ms β 482.29 ms β no change β
β Q9 β 531.76 ms β 522.68 ms β no change β
β Q10 β 1240.01 ms β 1238.87 ms β no change β
β Q11 β 374.76 ms β 375.73 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4282.57ms β
β Total Time (spill-dedup-view-arrays) β 4246.93ms β
β Average Time (HEAD) β 428.26ms β
β Average Time (spill-dedup-view-arrays) β 424.69ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.88 / 136.43 Β±1.69 / 139.73 ms β 134.90 / 135.61 Β±1.05 / 137.69 ms β no change β
β Q2 β 139.37 / 139.88 Β±0.50 / 140.83 ms β 137.61 / 139.40 Β±1.30 / 140.97 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.65 / 204.22 Β±3.78 / 208.97 ms β 200.15 / 201.52 Β±0.85 / 202.35 ms β no change β
β Q5 β 233.11 / 235.77 Β±1.78 / 238.35 ms β 228.56 / 232.89 Β±2.81 / 237.03 ms β no change β
β Q6 β 256.76 / 258.86 Β±1.33 / 260.61 ms β 252.52 / 258.45 Β±5.58 / 268.57 ms β no change β
β Q7 β 695.03 / 707.94 Β±9.72 / 721.34 ms β 673.63 / 693.70 Β±11.53 / 704.89 ms β no change β
β Q8 β 478.25 / 484.52 Β±5.37 / 490.45 ms β 482.29 / 486.87 Β±4.78 / 494.01 ms β no change β
β Q9 β 531.76 / 541.49 Β±5.36 / 546.65 ms β 522.68 / 532.48 Β±7.76 / 546.03 ms β no change β
β Q10 β 1240.01 / 1248.00 Β±8.07 / 1262.07 ms β 1238.87 / 1246.22 Β±5.74 / 1254.03 ms β no change β
β Q11 β 374.76 / 380.69 Β±4.38 / 387.94 ms β 375.73 / 378.63 Β±3.58 / 385.51 ms β no change β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4337.81ms β
β Total Time (spill-dedup-view-arrays) β 4305.77ms β
β Average Time (HEAD) β 433.78ms β
β Average Time (spill-dedup-view-arrays) β 430.58ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `7570366fd929daf9ced744bb8397686b50565b18` | Changed: `7570366fd929daf9ced744bb8397686b50565b18`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.5 MiB | 175.1 MiB | -0.2% |
| 2 | 220.6 MiB | 220.4 MiB | -0.1% |
| 3 | 524.1 MiB | 552.2 MiB | +5.4% |
| 4 | 265.8 MiB | 265.8 MiB | -0.0% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 356.5 MiB | 356.5 MiB | +0.0% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 497.9 MiB | +0.0% |
| 10 | 510.1 MiB | 506.5 MiB | -0.7% |
| 11 | 526.0 MiB | 523.6 MiB | -0.4% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`7570366fd929daf9ced744bb8397686b50565b18`) | 526.0 MiB | 1.3 GiB | 817.9 MiB | 2.6Γ |
| sort_tpch | changed (`7570366fd929daf9ced744bb8397686b50565b18`) | 552.2 MiB | 1.5 GiB | 987.1 MiB | 2.8Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 913.3 MiB |
| CPU user | 65.5s |
| CPU sys | 16.3s |
| Peak spill | 1.2 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.5 GiB |
| Avg memory | 967.5 MiB |
| CPU user | 63.6s |
| CPU sys | 14.9s |
| Peak spill | 1.1 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789851638
run benchmark spill_views sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
baseline:
ref: 7570366fd929daf9ced744bb8397686b50565b18
changed:
ref: 7570366fd929daf9ced744bb8397686b50565b18
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969212184
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpcds_sf1.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 5.47 / 5.98 Β±0.88 / 7.75 ms β 5.60 / 6.13 Β±0.85 / 7.82 ms β no change β
β QQuery 2 β 80.96 / 81.51 Β±0.44 / 82.19 ms β 80.62 / 81.03 Β±0.28 / 81.36 ms β no change β
β QQuery 3 β 29.29 / 29.53 Β±0.15 / 29.70 ms β 29.14 / 29.43 Β±0.16 / 29.62 ms β no change β
β QQuery 4 β 475.66 / 480.38 Β±5.78 / 489.39 ms β 485.60 / 488.82 Β±4.76 / 498.17 ms β no change β
β QQuery 5 β 51.65 / 52.10 Β±0.33 / 52.60 ms β 52.18 / 52.59 Β±0.34 / 53.12 ms β no change β
β QQuery 6 β 36.81 / 37.28 Β±0.46 / 38.17 ms β 37.05 / 37.20 Β±0.17 / 37.45 ms β no change β
β QQuery 7 β 94.14 / 94.63 Β±0.35 / 95.18 ms β 94.21 / 94.79 Β±0.38 / 95.29 ms β no change β
β QQuery 8 β 36.90 / 37.29 Β±0.34 / 37.90 ms β 37.08 / 37.57 Β±0.55 / 38.62 ms β no change β
β QQuery 9 β 52.74 / 54.35 Β±1.43 / 56.29 ms β 54.73 / 55.23 Β±0.42 / 56.00 ms β no change β
β QQuery 10 β 62.62 / 62.97 Β±0.32 / 63.38 ms β 63.03 / 63.39 Β±0.23 / 63.71 ms β no change β
β QQuery 11 β 293.29 / 295.79 Β±1.52 / 297.29 ms β 292.55 / 295.59 Β±1.75 / 297.54 ms β no change β
β QQuery 12 β 28.56 / 28.93 Β±0.33 / 29.53 ms β 28.40 / 28.77 Β±0.50 / 29.72 ms β no change β
β QQuery 13 β 118.39 / 119.23 Β±0.59 / 120.01 ms β 119.05 / 120.06 Β±0.79 / 120.94 ms β no change β
β QQuery 14 β 414.53 / 419.74 Β±4.20 / 425.27 ms β 418.00 / 420.22 Β±2.50 / 424.67 ms β no change β
β QQuery 15 β 57.30 / 58.17 Β±0.61 / 59.10 ms β 57.35 / 58.04 Β±0.60 / 59.17 ms β no change β
β QQuery 16 β 6.78 / 6.89 Β±0.15 / 7.17 ms β 6.83 / 6.96 Β±0.12 / 7.17 ms β no change β
β QQuery 17 β 80.64 / 81.68 Β±1.31 / 84.21 ms β 81.48 / 82.23 Β±0.93 / 84.03 ms β no change β
β QQuery 18 β 124.33 / 125.52 Β±1.47 / 128.35 ms β 124.33 / 124.98 Β±0.61 / 125.96 ms β no change β
β QQuery 19 β 41.28 / 41.51 Β±0.16 / 41.71 ms β 41.74 / 42.10 Β±0.35 / 42.62 ms β no change β
β QQuery 20 β 35.46 / 35.86 Β±0.28 / 36.17 ms β 35.54 / 35.98 Β±0.29 / 36.34 ms β no change β
β QQuery 21 β 17.41 / 17.72 Β±0.17 / 17.89 ms β 17.52 / 17.79 Β±0.23 / 18.07 ms β no change β
β QQuery 22 β 62.37 / 63.71 Β±1.48 / 66.54 ms β 62.46 / 63.76 Β±2.02 / 67.75 ms β no change β
β QQuery 23 β 347.33 / 348.96 Β±1.26 / 351.07 ms β 343.44 / 347.81 Β±3.50 / 354.13 ms β no change β
β QQuery 24 β 225.96 / 229.60 Β±5.94 / 241.39 ms β 226.43 / 229.68 Β±2.35 / 232.42 ms β no change β
β QQuery 25 β 111.08 / 112.04 Β±0.89 / 113.73 ms β 111.19 / 112.31 Β±1.03 / 113.88 ms β no change β
β QQuery 26 β 57.66 / 59.07 Β±2.17 / 63.39 ms β 57.63 / 59.87 Β±3.34 / 66.36 ms β no change β
β QQuery 27 β 6.28 / 6.41 Β±0.15 / 6.70 ms β 6.28 / 6.46 Β±0.21 / 6.86 ms β no change β
β QQuery 28 β 61.46 / 62.21 Β±0.71 / 63.42 ms β 61.97 / 62.63 Β±0.86 / 64.32 ms β no change β
β QQuery 29 β 97.33 / 98.19 Β±1.08 / 100.17 ms β 97.77 / 99.77 Β±2.18 / 103.61 ms β no change β
β QQuery 30 β 32.25 / 34.38 Β±2.99 / 40.23 ms β 32.51 / 33.45 Β±0.75 / 34.65 ms β no change β
β QQuery 31 β 110.69 / 112.33 Β±1.72 / 115.50 ms β 112.24 / 113.11 Β±0.48 / 113.67 ms β no change β
β QQuery 32 β 20.50 / 20.66 Β±0.27 / 21.20 ms β 20.33 / 20.68 Β±0.33 / 21.29 ms β no change β
β QQuery 33 β 37.91 / 38.31 Β±0.37 / 38.95 ms β 37.95 / 40.06 Β±3.84 / 47.73 ms β no change β
β QQuery 34 β 9.85 / 10.01 Β±0.18 / 10.35 ms β 9.82 / 10.01 Β±0.21 / 10.40 ms β no change β
β QQuery 35 β 73.03 / 74.88 Β±1.99 / 78.68 ms β 73.21 / 73.97 Β±0.40 / 74.34 ms β no change β
β QQuery 36 β 5.94 / 6.09 Β±0.15 / 6.37 ms β 5.88 / 6.04 Β±0.18 / 6.40 ms β no change β
β QQuery 37 β 6.92 / 7.05 Β±0.06 / 7.11 ms β 6.98 / 7.11 Β±0.12 / 7.32 ms β no change β
β QQuery 38 β 62.21 / 62.80 Β±0.30 / 63.08 ms β 62.92 / 63.11 Β±0.15 / 63.34 ms β no change β
β QQuery 39 β 90.53 / 90.91 Β±0.39 / 91.58 ms β 91.18 / 92.82 Β±1.83 / 96.35 ms β no change β
β QQuery 40 β 23.21 / 24.50 Β±0.70 / 25.06 ms β 23.40 / 23.80 Β±0.27 / 24.13 ms β no change β
β QQuery 41 β 11.57 / 11.71 Β±0.17 / 12.04 ms β 11.57 / 11.77 Β±0.20 / 12.15 ms β no change β
β QQuery 42 β 23.72 / 24.08 Β±0.29 / 24.60 ms β 23.92 / 24.11 Β±0.15 / 24.36 ms β no change β
β QQuery 43 β 5.04 / 5.11 Β±0.09 / 5.28 ms β 5.14 / 5.25 Β±0.13 / 5.50 ms β no change β
β QQuery 44 β 9.23 / 9.42 Β±0.11 / 9.57 ms β 9.70 / 9.76 Β±0.06 / 9.84 ms β no change β
β QQuery 45 β 38.87 / 39.17 Β±0.19 / 39.42 ms β 38.53 / 38.96 Β±0.25 / 39.25 ms β no change β
β QQuery 46 β 11.69 / 11.97 Β±0.20 / 12.14 ms β 11.94 / 12.14 Β±0.11 / 12.25 ms β no change β
β QQuery 47 β 227.08 / 229.72 Β±1.95 / 232.66 ms β 225.44 / 227.67 Β±2.41 / 232.34 ms β no change β
β QQuery 48 β 96.61 / 98.03 Β±1.53 / 100.87 ms β 96.65 / 97.55 Β±0.87 / 99.21 ms β no change β
β QQuery 49 β 76.30 / 77.23 Β±1.29 / 79.78 ms β 77.04 / 77.97 Β±0.60 / 78.65 ms β no change β
β QQuery 50 β 58.36 / 59.50 Β±0.70 / 60.25 ms β 58.98 / 62.57 Β±3.87 / 69.80 ms β 1.05x slower β
β QQuery 51 β 90.98 / 92.89 Β±2.20 / 96.62 ms β 92.43 / 94.43 Β±1.23 / 96.17 ms β no change β
β QQuery 52 β 23.48 / 23.94 Β±0.25 / 24.25 ms β 24.16 / 24.42 Β±0.29 / 24.97 ms β no change β
β QQuery 53 β 29.31 / 29.64 Β±0.28 / 30.13 ms β 30.33 / 30.59 Β±0.30 / 31.17 ms β no change β
β QQuery 54 β 55.08 / 57.50 Β±3.44 / 64.32 ms β 56.63 / 59.05 Β±2.98 / 64.81 ms β no change β
β QQuery 55 β 23.14 / 23.53 Β±0.57 / 24.66 ms β 24.27 / 24.79 Β±0.31 / 25.16 ms β 1.05x slower β
β QQuery 56 β 39.19 / 39.89 Β±0.58 / 40.90 ms β 39.72 / 40.17 Β±0.50 / 41.13 ms β no change β
β QQuery 57 β 176.55 / 180.83 Β±3.85 / 188.04 ms β 178.98 / 181.17 Β±2.32 / 185.42 ms β no change β
β QQuery 58 β 115.91 / 117.99 Β±1.62 / 120.84 ms β 116.98 / 119.13 Β±2.41 / 123.58 ms β no change β
β QQuery 59 β 119.38 / 120.32 Β±1.13 / 121.98 ms β 119.29 / 120.52 Β±0.70 / 121.36 ms β no change β
β QQuery 60 β 39.38 / 40.29 Β±0.65 / 41.21 ms β 40.08 / 40.57 Β±0.65 / 41.81 ms β no change β
β QQuery 61 β 12.86 / 13.00 Β±0.20 / 13.39 ms β 12.91 / 13.09 Β±0.14 / 13.33 ms β no change β
β QQuery 62 β 46.65 / 48.11 Β±2.37 / 52.82 ms β 47.21 / 48.35 Β±1.90 / 52.14 ms β no change β
β QQuery 63 β 30.18 / 30.79 Β±0.61 / 31.73 ms β 30.53 / 31.74 Β±1.52 / 34.74 ms β no change β
β QQuery 64 β 408.48 / 415.90 Β±7.68 / 429.92 ms β 419.87 / 421.64 Β±1.59 / 424.32 ms β no change β
β QQuery 65 β 145.49 / 149.23 Β±2.92 / 153.88 ms β 146.68 / 149.17 Β±1.54 / 150.58 ms β no change β
β QQuery 66 β 79.88 / 82.67 Β±3.89 / 90.37 ms β 81.41 / 82.44 Β±0.97 / 84.26 ms β no change β
β QQuery 67 β 236.24 / 241.32 Β±4.47 / 247.68 ms β 239.08 / 242.90 Β±2.52 / 246.03 ms β no change β
β QQuery 68 β 11.87 / 12.07 Β±0.26 / 12.58 ms β 12.13 / 12.27 Β±0.19 / 12.64 ms β no change β
β QQuery 69 β 57.04 / 57.63 Β±0.54 / 58.56 ms β 58.74 / 60.35 Β±2.82 / 65.99 ms β no change β
β QQuery 70 β 106.38 / 111.64 Β±7.32 / 125.95 ms β 106.27 / 110.01 Β±4.45 / 117.87 ms β no change β
β QQuery 71 β 35.71 / 36.03 Β±0.26 / 36.49 ms β 36.06 / 36.31 Β±0.16 / 36.47 ms β no change β
β QQuery 72 β 2090.46 / 2210.07 Β±80.08 / 2310.63 ms β 2163.70 / 2219.16 Β±48.84 / 2298.38 ms β no change β
β QQuery 73 β 9.55 / 9.82 Β±0.25 / 10.25 ms β 9.66 / 12.36 Β±3.92 / 19.98 ms β 1.26x slower β
β QQuery 74 β 168.61 / 172.01 Β±3.12 / 177.72 ms β 168.68 / 171.05 Β±2.22 / 173.97 ms β no change β
β QQuery 75 β 149.63 / 152.55 Β±4.16 / 160.80 ms β 149.87 / 151.10 Β±0.85 / 152.34 ms β no change β
β QQuery 76 β 35.55 / 36.18 Β±0.52 / 36.88 ms β 35.31 / 35.82 Β±0.26 / 36.01 ms β no change β
β QQuery 77 β 61.76 / 62.13 Β±0.23 / 62.40 ms β 61.58 / 62.02 Β±0.45 / 62.87 ms β no change β
β QQuery 78 β 197.32 / 200.01 Β±2.10 / 202.70 ms β 195.17 / 200.98 Β±5.61 / 209.00 ms β no change β
β QQuery 79 β 67.50 / 72.05 Β±6.50 / 84.64 ms β 67.75 / 69.85 Β±3.53 / 76.88 ms β no change β
β QQuery 80 β 99.99 / 101.59 Β±1.63 / 103.82 ms β 99.49 / 100.93 Β±1.24 / 102.96 ms β no change β
β QQuery 81 β 25.63 / 25.91 Β±0.17 / 26.15 ms β 25.96 / 28.24 Β±3.72 / 35.66 ms β 1.09x slower β
β QQuery 82 β 16.29 / 16.76 Β±0.25 / 17.03 ms β 17.27 / 18.93 Β±1.85 / 21.72 ms β 1.13x slower β
β QQuery 83 β 40.66 / 44.96 Β±4.96 / 54.29 ms β 40.60 / 41.08 Β±0.65 / 42.34 ms β +1.09x faster β
β QQuery 84 β 30.85 / 31.18 Β±0.21 / 31.45 ms β 30.88 / 31.07 Β±0.10 / 31.15 ms β no change β
β QQuery 85 β 107.28 / 108.31 Β±0.80 / 109.31 ms β 107.16 / 108.96 Β±1.49 / 111.69 ms β no change β
β QQuery 86 β 25.24 / 25.45 Β±0.27 / 25.90 ms β 24.94 / 26.25 Β±1.00 / 27.93 ms β no change β
β QQuery 87 β 63.38 / 65.58 Β±2.74 / 70.82 ms β 63.16 / 63.95 Β±0.82 / 65.48 ms β no change β
β QQuery 88 β 62.82 / 63.50 Β±0.55 / 64.15 ms β 63.80 / 64.33 Β±0.48 / 65.03 ms β no change β
β QQuery 89 β 35.59 / 35.96 Β±0.22 / 36.19 ms β 35.57 / 37.12 Β±1.92 / 40.88 ms β no change β
β QQuery 90 β 17.20 / 17.37 Β±0.16 / 17.67 ms β 17.37 / 17.55 Β±0.20 / 17.89 ms β no change β
β QQuery 91 β 45.89 / 47.22 Β±1.15 / 49.24 ms β 46.77 / 47.58 Β±0.66 / 48.67 ms β no change β
β QQuery 92 β 29.03 / 29.74 Β±0.75 / 31.16 ms β 29.29 / 29.90 Β±0.72 / 31.27 ms β no change β
β QQuery 93 β 49.75 / 50.47 Β±0.79 / 51.93 ms β 50.03 / 50.64 Β±0.47 / 51.46 ms β no change β
β QQuery 94 β 38.22 / 38.91 Β±0.50 / 39.67 ms β 38.28 / 38.70 Β±0.34 / 39.21 ms β no change β
β QQuery 95 β 81.96 / 83.58 Β±1.69 / 86.66 ms β 82.66 / 84.66 Β±1.75 / 86.85 ms β no change β
β QQuery 96 β 24.39 / 24.71 Β±0.29 / 25.14 ms β 24.47 / 24.63 Β±0.15 / 24.89 ms β no change β
β QQuery 97 β 46.64 / 47.40 Β±0.62 / 48.39 ms β 47.06 / 47.37 Β±0.25 / 47.61 ms β no change β
β QQuery 98 β 42.10 / 42.91 Β±0.44 / 43.35 ms β 42.63 / 43.44 Β±0.50 / 43.92 ms β no change β
β QQuery 99 β 70.09 / 70.46 Β±0.32 / 71.05 ms β 69.93 / 71.81 Β±2.11 / 75.67 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 10007.10ms β
β Total Time (spill-dedup-view-arrays) β 10055.70ms β
β Average Time (HEAD) β 101.08ms β
β Average Time (spill-dedup-view-arrays) β 101.57ms β
β Queries Faster β 1 β
β Queries Slower β 5 β
β Queries with No Change β 93 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.0 GiB |
| Avg memory | 1.3 GiB |
| CPU user | 228.6s |
| CPU sys | 5.8s |
| Peak spill | 0 B |
**tpcds β branch**
| Metric | Value |
|--------|-------|
| Wall time | 55.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.5 GiB |
| CPU user | 222.2s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783343463
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783081490)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783081490-2592-dt4vt 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784344459
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784305266)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5784305266-2615-h2pwd 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789161668
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789125588)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789125588-2627-4sjcw 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789376299
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789122907)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 105.3Β±2.61ms ? ?/sec 1.09 114.5Β±0.71ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 101.1Β±2.92ms ? ?/sec 1.00 101.4Β±3.64ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 98.0Β±0.70ms ? ?/sec 1.00 98.5Β±2.02ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 125.5Β±1.58ms ? ?/sec 1.03 128.7Β±5.10ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.04 115.6Β±4.31ms ? ?/sec 1.00 111.6Β±1.01ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 560.3s |
| Peak memory | 852.7 MiB |
| Avg memory | 40.5 MiB |
| CPU user | 66.5s |
| CPU sys | 29.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 580.3s |
| Peak memory | 803.3 MiB |
| Avg memory | 35.5 MiB |
| CPU user | 65.8s |
| CPU sys | 29.7s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783120747
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783075232-2586-455hd 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783902036
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783727326)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.42 ms β 134.27 ms β no change β
β Q2 β 135.07 ms β 135.66 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 197.03 ms β 197.53 ms β no change β
β Q5 β 228.98 ms β 226.36 ms β no change β
β Q6 β 249.96 ms β 250.39 ms β no change β
β Q7 β 692.13 ms β 682.24 ms β no change β
β Q8 β 476.83 ms β 507.85 ms β 1.07x slower β
β Q9 β 533.15 ms β 558.70 ms β no change β
β Q10 β 1232.09 ms β 1245.78 ms β no change β
β Q11 β 375.72 ms β 405.84 ms β 1.08x slower β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4255.38ms β
β Total Time (spill-dedup-view-arrays) β 4344.61ms β
β Average Time (HEAD) β 425.54ms β
β Average Time (spill-dedup-view-arrays) β 434.46ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 8 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.42 / 135.99 Β±1.14 / 137.80 ms β 134.27 / 135.05 Β±0.70 / 136.16 ms β no change β
β Q2 β 135.07 / 136.26 Β±1.27 / 138.27 ms β 135.66 / 137.09 Β±1.29 / 139.01 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 197.03 / 201.13 Β±2.52 / 204.23 ms β 197.53 / 203.35 Β±4.20 / 209.51 ms β no change β
β Q5 β 228.98 / 230.62 Β±1.94 / 234.35 ms β 226.36 / 230.17 Β±2.00 / 231.66 ms β no change β
β Q6 β 249.96 / 252.25 Β±2.13 / 255.28 ms β 250.39 / 252.08 Β±1.19 / 253.51 ms β no change β
β Q7 β 692.13 / 695.54 Β±2.69 / 698.20 ms β 682.24 / 691.20 Β±6.35 / 700.25 ms β no change β
β Q8 β 476.83 / 485.60 Β±7.85 / 497.89 ms β 507.85 / 513.03 Β±4.52 / 520.77 ms β 1.06x slower β
β Q9 β 533.15 / 535.83 Β±3.02 / 541.16 ms β 558.70 / 565.09 Β±5.65 / 573.12 ms β 1.05x slower β
β Q10 β 1232.09 / 1243.47 Β±6.72 / 1250.75 ms β 1245.78 / 1261.33 Β±14.90 / 1286.00 ms β no change β
β Q11 β 375.72 / 377.73 Β±1.95 / 381.16 ms β 405.84 / 409.82 Β±2.95 / 414.57 ms β 1.08x slower β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4294.41ms β
β Total Time (spill-dedup-view-arrays) β 4398.21ms β
β Average Time (HEAD) β 429.44ms β
β Average Time (spill-dedup-view-arrays) β 439.82ms β
β Queries Faster β 0 β
β Queries Slower β 3 β
β Queries with No Change β 7 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `5b5bed9 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 174.9 MiB | 175.3 MiB | +0.2% |
| 2 | 220.1 MiB | 219.4 MiB | -0.3% |
| 3 | 537.8 MiB | 515.8 MiB | -4.1% |
| 4 | 262.7 MiB | 265.5 MiB | +1.1% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 356.5 MiB | 356.1 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 505.7 MiB | 497.9 MiB | -1.5% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 523.4 MiB | 523.2 MiB | -0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`5b5bed9 (merge-base)`) | 537.8 MiB | 1.3 GiB | 754.1 MiB | 2.4Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 523.2 MiB | 1.1 GiB | 617.7 MiB | 2.2Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 907.2 MiB |
| CPU user | 69.5s |
| CPU sys | 16.5s |
| Peak spill | 1.1 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 848.6 MiB |
| CPU user | 66.5s |
| CPU sys | 15.8s |
| Peak spill | 1.0 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783928487
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783901920)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783901920-2609-2k6d5 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r3755018057
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -420,6 +420,20 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Garbage collect and deduplicate a string view before writing it out to disk
+///
+/// This is to avoid inflating disk usage and also to ensure that deduplication reduces memory pressure when reading back.
+fn gc_dedup_view<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> GenericByteViewArray<T> {
+ let mut builder = GenericByteViewBuilder::<T>::with_capacity(array.len())
Review Comment:
Yes this is the trade off. However, I believe the size of this is actually dictated by the configured batch size so it's never going to be more than that.
So with `16384` as the default batch size this could use ~ `16384 * 64 = 1MiB <+ hash internals>` RAM to write this out + CPU overhead of hash lookups which I think is ~ `log(n)`. So yeah high cardinality string views will probably cause increased memory/cpu.
In high cardinality scenarios this will add overhead, but it's hard to measure without doing a second pass to work out what the cardinality is. Maybe there is a better way to gauge the cardinality of the input view and use that to decide? Not sure there is a good method for this.
But, in production we have seen a good reduction in file usage/memory pressure with gc & dedup (i.e, we have applied this as a patch to our version of DF), so I feel that for real use cases the trade off is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248861843
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5249076127
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5249003089)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 150.39 ms β 151.06 ms β no change β
β Q2 β 135.83 ms β 136.40 ms β no change β
β Q3 β 602.63 ms β 699.14 ms β 1.16x slower β
β Q4 β 249.09 ms β 252.54 ms β no change β
β Q5 β 264.93 ms β 267.72 ms β no change β
β Q6 β 290.46 ms β 292.75 ms β no change β
β Q7 β 725.17 ms β 734.68 ms β no change β
β Q8 β 503.43 ms β 528.52 ms β no change β
β Q9 β 540.82 ms β 565.44 ms β no change β
β Q10 β 1227.66 ms β 1263.74 ms β no change β
β Q11 β 425.69 ms β 454.23 ms β 1.07x slower β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 5116.11ms β
β Total Time (spill-dedup-view-arrays) β 5346.22ms β
β Average Time (HEAD) β 465.10ms β
β Average Time (spill-dedup-view-arrays) β 486.02ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 9 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 150.39 / 151.27 Β±1.12 / 153.35 ms β 151.06 / 151.79 Β±0.92 / 153.55 ms β no change β
β Q2 β 135.83 / 138.62 Β±2.36 / 142.10 ms β 136.40 / 139.36 Β±3.03 / 143.18 ms β no change β
β Q3 β 602.63 / 613.91 Β±6.65 / 622.24 ms β 699.14 / 706.24 Β±7.39 / 720.05 ms β 1.15x slower β
β Q4 β 249.09 / 256.85 Β±5.68 / 263.50 ms β 252.54 / 257.31 Β±4.92 / 266.73 ms β no change β
β Q5 β 264.93 / 268.12 Β±2.14 / 270.99 ms β 267.72 / 271.01 Β±1.86 / 273.09 ms β no change β
β Q6 β 290.46 / 295.84 Β±3.90 / 301.12 ms β 292.75 / 296.51 Β±2.50 / 300.19 ms β no change β
β Q7 β 725.17 / 735.03 Β±5.19 / 738.83 ms β 734.68 / 742.74 Β±5.68 / 750.06 ms β no change β
β Q8 β 503.43 / 506.24 Β±2.00 / 508.36 ms β 528.52 / 532.73 Β±3.08 / 537.99 ms β 1.05x slower β
β Q9 β 540.82 / 549.17 Β±7.11 / 559.09 ms β 565.44 / 572.55 Β±5.80 / 579.23 ms β no change β
β Q10 β 1227.66 / 1237.65 Β±8.35 / 1251.47 ms β 1263.74 / 1276.78 Β±12.75 / 1298.87 ms β no change β
β Q11 β 425.69 / 431.41 Β±3.38 / 436.28 ms β 454.23 / 457.47 Β±3.83 / 464.54 ms β 1.06x slower β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 5184.10ms β
β Total Time (spill-dedup-view-arrays) β 5404.49ms β
β Average Time (HEAD) β 471.28ms β
β Average Time (spill-dedup-view-arrays) β 491.32ms β
β Queries Faster β 0 β
β Queries Slower β 3 β
β Queries with No Change β 8 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `a9b61ab (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 174.9 MiB | 175.5 MiB | +0.3% |
| 2 | 219.9 MiB | 220.6 MiB | +0.3% |
| 3 | 505.0 MiB | 503.0 MiB | -0.4% |
| 4 | 266.1 MiB | 264.6 MiB | -0.6% |
| 5 | 295.0 MiB | 295.0 MiB | +0.0% |
| 6 | 356.1 MiB | 355.7 MiB | -0.1% |
| 7 | 509.0 MiB | 506.4 MiB | -0.5% |
| 8 | 505.8 MiB | 504.1 MiB | -0.3% |
| 9 | 505.1 MiB | 505.1 MiB | +0.0% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 508.9 MiB | 508.9 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`a9b61ab (merge-base)`) | 510.1 MiB | 1.8 GiB | 1.3 GiB | 3.7Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 510.1 MiB | 1.5 GiB | 990.1 MiB | 2.9Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 30.0s |
| Peak memory | 1.8 GiB |
| Avg memory | 987.4 MiB |
| CPU user | 79.9s |
| CPU sys | 16.6s |
| Peak spill | 0 B |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 30.0s |
| Peak memory | 1.5 GiB |
| Avg memory | 1022.6 MiB |
| CPU user | 83.0s |
| CPU sys | 17.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863139585
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111096)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863111096-2862-czgxs 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863232955
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111096)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 136.07 ms β 135.72 ms β no change β
β Q2 β 136.02 ms β 134.82 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 200.20 ms β 197.38 ms β no change β
β Q5 β 232.60 ms β 233.58 ms β no change β
β Q6 β 253.75 ms β 254.25 ms β no change β
β Q7 β 685.21 ms β 680.83 ms β no change β
β Q8 β 487.02 ms β 479.01 ms β no change β
β Q9 β 534.12 ms β 531.20 ms β no change β
β Q10 β 1266.16 ms β 1251.16 ms β no change β
β Q11 β 371.74 ms β 375.42 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4302.89ms β
β Total Time (spill-dedup-view-arrays) β 4273.39ms β
β Average Time (HEAD) β 430.29ms β
β Average Time (spill-dedup-view-arrays) β 427.34ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 136.07 / 137.50 Β±1.39 / 140.14 ms β 135.72 / 136.50 Β±1.03 / 138.37 ms β no change β
β Q2 β 136.02 / 137.18 Β±1.02 / 138.85 ms β 134.82 / 136.52 Β±1.67 / 139.24 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 200.20 / 204.21 Β±3.91 / 209.51 ms β 197.38 / 204.46 Β±5.38 / 212.19 ms β no change β
β Q5 β 232.60 / 234.09 Β±1.18 / 235.99 ms β 233.58 / 235.39 Β±1.59 / 237.88 ms β no change β
β Q6 β 253.75 / 256.67 Β±1.92 / 258.29 ms β 254.25 / 256.26 Β±1.92 / 258.98 ms β no change β
β Q7 β 685.21 / 702.46 Β±14.47 / 727.65 ms β 680.83 / 692.94 Β±8.01 / 703.31 ms β no change β
β Q8 β 487.02 / 490.64 Β±2.73 / 494.01 ms β 479.01 / 485.65 Β±5.34 / 491.33 ms β no change β
β Q9 β 534.12 / 536.45 Β±2.00 / 539.55 ms β 531.20 / 537.98 Β±6.32 / 549.27 ms β no change β
β Q10 β 1266.16 / 1272.62 Β±5.66 / 1281.88 ms β 1251.16 / 1264.59 Β±10.32 / 1279.02 ms β no change β
β Q11 β 371.74 / 382.04 Β±7.00 / 393.54 ms β 375.42 / 379.37 Β±4.21 / 387.35 ms β no change β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4353.85ms β
β Total Time (spill-dedup-view-arrays) β 4329.65ms β
β Average Time (HEAD) β 435.39ms β
β Average Time (spill-dedup-view-arrays) β 432.97ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69fdca8737c9d065b53b5fc5e4183302775` | Changed: `ed43e69fdca8737c9d065b53b5fc5e4183302775`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 174.3 MiB | 174.7 MiB | +0.2% |
| 2 | 219.9 MiB | 220.4 MiB | +0.2% |
| 3 | 516.0 MiB | 513.7 MiB | -0.5% |
| 4 | 261.4 MiB | 263.3 MiB | +0.7% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.6 MiB | 354.4 MiB | -0.4% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 498.2 MiB | 502.3 MiB | +0.8% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 523.8 MiB | 522.8 MiB | -0.2% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 523.8 MiB | 1.3 GiB | 761.9 MiB | 2.5Γ |
| sort_tpch | changed (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 522.8 MiB | 1.2 GiB | 729.4 MiB | 2.4Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 876.2 MiB |
| CPU user | 66.0s |
| CPU sys | 16.5s |
| Peak spill | 968.7 MiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 857.5 MiB |
| CPU user | 63.2s |
| CPU sys | 15.5s |
| Peak spill | 1.1 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r4118923989
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -800,29 +804,133 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Compacts the data buffers of a view array, or returns `None` when the
+/// array is too small for compaction to be worth it.
+///
+/// Arrow's `gc()` copies the bytes of every view separately, so repeated
+/// values each get their own copy. For a dictionary-encoded Parquet column
+/// this inflates the spilled data by the average repeat count: 1M rows of
+/// 1000 distinct 64 byte values spill as 64 MB
+/// (<https://github.com/apache/datafusion/issues/23564>).
+/// [`gc_dedup_view_array`] copies each distinct value once instead, and falls
+/// back to `gc()` when the values turn out to be distinct.
+fn gc_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ if !should_gc_view_array(array) {
+ return None;
+ }
+ Some(gc_dedup_view_array(array).unwrap_or_else(|| array.gc()))
+}
+
+/// Number of non-inline values [`gc_dedup_view_array`] deduplicates before
+/// it checks whether the array has enough repeats to be worth it.
+const DEDUP_SAMPLE_VALUES: usize = 256;
+
+/// Like `gc()`, but copies each distinct value once, and zeroes null views.
+///
+/// Returns `None` if the first [`DEDUP_SAMPLE_VALUES`] non-inline values
+/// hold almost no repeats, since hashing every value then costs more than
+/// deduplication saves.
Review Comment:
The sample turns dedup off only when the first 256 non-inline values have fewer than 4 repeats (1.6%). If a batch has a few repeats but is mostly distinct, dedup hashes every value and saves few bytes. The `spill_views` suite has no query for this case: q02 (1000 distinct values) is above the threshold and saves 88% of the bytes, and q04 (all distinct) is below it.
I ran the q04 shape (1M rows, 64-byte payload, `ORDER BY id`, 96M limit, 4 partitions) with part of the rows set to one repeated value. `datafusion-cli` at merge-base `ed43e69` vs `7effeab`, 25 interleaved runs per side, Apple silicon laptop:
| Data | PR / base (median of paired ratios) | PR slower in | Spilled bytes |
| --- | --- | --- | --- |
| All distinct (control) | 1.06x | 13 of 25 | 84.2 β 84.2 MB |
| 5% one repeated value | **1.52x** | 23 of 25 | 84.2 β 81.1 MB |
| 20% one repeated value | **1.32x** | 21 of 25 | 84.2 β 72.0 MB |
| 1000 distinct values (40M limit) | 0.88x | 8 of 25 | 84.2 β 30.6 MB |
The control moves by about Β±10% on this machine. An earlier 9-run pass gave 1.39x (5%) and 1.37x (20%). This agrees with q04 in the round before the sample (1.38β1.45x), when dedup always ran.
Could the decision use the bytes that dedup saves in the sample, instead of the count of repeats? For example: sum the byte length of the sampled values and of the distinct sampled values, and fall back to `gc()` when the saving is below a threshold. Please set the threshold from a measured sweep of the repeated fraction (for example 5%, 20%, 50%, 80%). On my laptop the break-even is between 20% saved (1.32x slower) and 88% saved (0.88x).
<details><summary>Reproducer (5% case)</summary>
```sql
set datafusion.execution.target_partitions = 4;
COPY (
SELECT (value * 7919) % 1000003 AS id,
'payload-' || lpad(CAST(CASE WHEN (value * 2654435761) % 100 < 5 THEN 0 ELSE value END AS VARCHAR), 56, '0') AS s
FROM generate_series(1, 1000000)
) TO 'hot5.parquet' STORED AS PARQUET;
set datafusion.runtime.memory_limit = '96M';
CREATE EXTERNAL TABLE t STORED AS PARQUET LOCATION 'hot5.parquet';
EXPLAIN ANALYZE SELECT id, s FROM t ORDER BY id;
```
For 20%, change `< 5` to `< 20`.
</details>
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -800,29 +804,133 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Compacts the data buffers of a view array, or returns `None` when the
+/// array is too small for compaction to be worth it.
+///
+/// Arrow's `gc()` copies the bytes of every view separately, so repeated
+/// values each get their own copy. For a dictionary-encoded Parquet column
+/// this inflates the spilled data by the average repeat count: 1M rows of
+/// 1000 distinct 64 byte values spill as 64 MB
+/// (<https://github.com/apache/datafusion/issues/23564>).
+/// [`gc_dedup_view_array`] copies each distinct value once instead, and falls
+/// back to `gc()` when the values turn out to be distinct.
+fn gc_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ if !should_gc_view_array(array) {
+ return None;
+ }
+ Some(gc_dedup_view_array(array).unwrap_or_else(|| array.gc()))
+}
+
+/// Number of non-inline values [`gc_dedup_view_array`] deduplicates before
+/// it checks whether the array has enough repeats to be worth it.
+const DEDUP_SAMPLE_VALUES: usize = 256;
+
+/// Like `gc()`, but copies each distinct value once, and zeroes null views.
+///
+/// Returns `None` if the first [`DEDUP_SAMPLE_VALUES`] non-inline values
+/// hold almost no repeats, since hashing every value then costs more than
+/// deduplication saves.
+fn gc_dedup_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ let buffers = array.data_buffers();
+ let bytes_of = |view: &ByteView| {
+ let start = view.offset as usize;
+ &buffers[view.buffer_index as usize][start..start + view.length as usize]
+ };
+ let hasher = DefaultHashBuilder::default();
+
+ // (input view, output view) of the first occurrence of each distinct value
+ let mut distinct: HashTable<(u128, u128)> = HashTable::new();
+ let mut completed: Vec<Buffer> = vec![];
+ let mut data: Vec<u8> = vec![];
+ let mut non_inline = 0;
+ let mut views = Vec::with_capacity(array.len());
+
+ for (i, &raw) in array.views().iter().enumerate() {
+ if array.is_null(i) {
+ views.push(0);
+ continue;
+ }
+ if (raw as u32) <= MAX_INLINE_VIEW_LEN {
+ views.push(raw);
+ continue;
+ }
+
+ let view = ByteView::from(raw);
+ let bytes = bytes_of(&view);
+ let hash = hasher.hash_one(bytes);
+ let found = distinct.find(hash, |(first, _)| {
+ *first == raw || bytes_of(&ByteView::from(*first)) == bytes
+ });
+ let new_view = match found {
+ Some((_, new_view)) => *new_view,
+ None => {
+ // A view offset is a `u32`, and a buffer may not exceed `i32::MAX`
+ if data.len() + bytes.len() > i32::MAX as usize {
+ completed.push(Buffer::from_vec(std::mem::take(&mut data)));
+ }
+ let new_view = ByteView {
+ buffer_index: completed.len() as u32,
+ offset: data.len() as u32,
+ ..view
+ }
+ .as_u128();
+ data.extend_from_slice(bytes);
+ distinct.insert_unique(hash, (raw, new_view), |(first, _)| {
+ hasher.hash_one(bytes_of(&ByteView::from(*first)))
+ });
+ new_view
+ }
+ };
+ views.push(new_view);
+
+ non_inline += 1;
+ if non_inline == DEDUP_SAMPLE_VALUES
+ && non_inline - distinct.len() < DEDUP_SAMPLE_VALUES / 64
+ {
+ return None;
+ }
Review Comment:
`distinct` and `data` start empty and grow while the loop runs. Each time the table grows, it hashes the bytes of every stored value again. In a microbenchmark of this function (8192-row batches, 64-byte values, views into one shared buffer), dedup takes 8β12x the time of `gc()` when the batch has few repeats, and 2β4x for 1 or 1000 distinct values.
This change reserves capacity after the sample passes, scaled by the distinct ratio of the sample. With it, the few-repeats cases drop to about 3.5β4x of `gc()`, and 1 or 1000 distinct values improve a little. The all-distinct case does not change, because the sample returns before the reserve. Do not reserve at the start: with a table sized for the full array, the all-distinct case went from 1.25x to 1.6x of `gc()`.
| Batch | This PR vs `gc()` | With the reserve |
| --- | --- | --- |
| 4 repeats in the sample, rest distinct | 10.1β10.8x | 3.4β3.7x |
| 5% one repeated value | 10.5β11.8x | 3.1β4.1x |
| 20% one repeated value | 8.1β8.6x | 4.0β5.1x |
| 1000 distinct values | 3.8β4.1x | 3.1β3.3x |
| 1 distinct value | 1.9β2.1x | 1.8x |
```suggestion
if non_inline == DEDUP_SAMPLE_VALUES {
if non_inline - distinct.len() < DEDUP_SAMPLE_VALUES / 64 {
return None;
}
// Size for the rest of the array from the sample's distinct ratio
let remaining = array.len() - i - 1;
let expected = remaining * distinct.len() / DEDUP_SAMPLE_VALUES;
distinct.reserve(expected, |(first, _)| {
hasher.hash_one(bytes_of(&ByteView::from(*first)))
});
data.reserve(expected * data.len() / distinct.len());
}
```
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -800,29 +804,133 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Compacts the data buffers of a view array, or returns `None` when the
+/// array is too small for compaction to be worth it.
+///
+/// Arrow's `gc()` copies the bytes of every view separately, so repeated
+/// values each get their own copy. For a dictionary-encoded Parquet column
+/// this inflates the spilled data by the average repeat count: 1M rows of
+/// 1000 distinct 64 byte values spill as 64 MB
+/// (<https://github.com/apache/datafusion/issues/23564>).
+/// [`gc_dedup_view_array`] copies each distinct value once instead, and falls
+/// back to `gc()` when the values turn out to be distinct.
+fn gc_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ if !should_gc_view_array(array) {
+ return None;
+ }
+ Some(gc_dedup_view_array(array).unwrap_or_else(|| array.gc()))
+}
+
+/// Number of non-inline values [`gc_dedup_view_array`] deduplicates before
+/// it checks whether the array has enough repeats to be worth it.
+const DEDUP_SAMPLE_VALUES: usize = 256;
+
+/// Like `gc()`, but copies each distinct value once, and zeroes null views.
+///
+/// Returns `None` if the first [`DEDUP_SAMPLE_VALUES`] non-inline values
+/// hold almost no repeats, since hashing every value then costs more than
+/// deduplication saves.
+fn gc_dedup_view_array<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> Option<GenericByteViewArray<T>> {
+ let buffers = array.data_buffers();
+ let bytes_of = |view: &ByteView| {
+ let start = view.offset as usize;
+ &buffers[view.buffer_index as usize][start..start + view.length as usize]
+ };
+ let hasher = DefaultHashBuilder::default();
+
+ // (input view, output view) of the first occurrence of each distinct value
+ let mut distinct: HashTable<(u128, u128)> = HashTable::new();
Review Comment:
This table is not reserved from the memory pool. `gc()` on `main` also allocates its output buffers without a reservation, so the new unaccounted memory is the table. Each entry is 32 bytes plus one control byte, and hashbrown rounds the bucket count up to a power of two. For an 8192-row batch that passes the sample with mostly distinct values, that is 16384 buckets Γ 33 bytes β 530 KiB. It is freed after each batch, but it grows with the rows per spilled batch, and each partition that spills at the same time holds one.
Could you either reserve it (for example through a `MemoryReservation` from the `SpillManager`), or add a comment here that states the bound and why a reservation is not necessary? A comment is enough for me if the rows per spilled batch are bounded for all callers of `gc_view_arrays`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863478233
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111096)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.01 108.6Β±0.95ms ? ?/sec 1.00 107.1Β±0.54ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.01 104.0Β±0.71ms ? ?/sec 1.00 102.9Β±0.77ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.02 102.3Β±0.70ms ? ?/sec 1.00 100.7Β±0.39ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.01 130.4Β±1.52ms ? ?/sec 1.00 128.7Β±1.53ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.02 137.3Β±3.44ms ? ?/sec 1.00 134.3Β±5.79ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69fdca8737c9d065b53b5fc5e4183302775` | Changed: `ed43e69fdca8737c9d065b53b5fc5e4183302775`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 102.6 MiB | 727.5 MiB | 624.9 MiB | 7.1Γ |
| spill_views | changed (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 102.6 MiB | 787.3 MiB | 684.8 MiB | 7.7Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 670.3s |
| Peak memory | 727.5 MiB |
| Avg memory | 32.7 MiB |
| CPU user | 63.7s |
| CPU sys | 30.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 670.4s |
| Peak memory | 787.3 MiB |
| Avg memory | 33.1 MiB |
| CPU user | 64.0s |
| CPU sys | 29.9s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867447282
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867396562)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5867396562-2874-ts6kj 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867549870
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867397520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.77 ms β 135.43 ms β no change β
β Q2 β 136.39 ms β 137.98 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.64 ms β 201.01 ms β no change β
β Q5 β 229.05 ms β 231.48 ms β no change β
β Q6 β 253.46 ms β 253.52 ms β no change β
β Q7 β 693.15 ms β 670.37 ms β no change β
β Q8 β 477.49 ms β 472.08 ms β no change β
β Q9 β 520.82 ms β 524.98 ms β no change β
β Q10 β 1221.39 ms β 1209.75 ms β no change β
β Q11 β 374.48 ms β 370.50 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4240.66ms β
β Total Time (spill-dedup-view-arrays) β 4207.08ms β
β Average Time (HEAD) β 424.07ms β
β Average Time (spill-dedup-view-arrays) β 420.71ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.77 / 136.60 Β±0.91 / 138.13 ms β 135.43 / 136.28 Β±1.34 / 138.92 ms β no change β
β Q2 β 136.39 / 137.38 Β±1.03 / 138.64 ms β 137.98 / 139.76 Β±1.27 / 141.91 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.64 / 204.05 Β±3.88 / 208.53 ms β 201.01 / 205.01 Β±3.40 / 210.60 ms β no change β
β Q5 β 229.05 / 232.24 Β±2.74 / 237.04 ms β 231.48 / 234.62 Β±1.82 / 236.51 ms β no change β
β Q6 β 253.46 / 256.68 Β±2.41 / 259.31 ms β 253.52 / 257.43 Β±2.75 / 261.46 ms β no change β
β Q7 β 693.15 / 696.44 Β±2.64 / 700.04 ms β 670.37 / 681.34 Β±9.61 / 696.97 ms β no change β
β Q8 β 477.49 / 487.06 Β±5.58 / 493.82 ms β 472.08 / 479.51 Β±5.29 / 486.68 ms β no change β
β Q9 β 520.82 / 528.81 Β±5.57 / 534.84 ms β 524.98 / 526.64 Β±1.28 / 528.48 ms β no change β
β Q10 β 1221.39 / 1234.24 Β±10.21 / 1252.06 ms β 1209.75 / 1230.46 Β±14.64 / 1247.69 ms β no change β
β Q11 β 374.48 / 376.96 Β±2.46 / 380.83 ms β 370.50 / 375.86 Β±5.44 / 384.93 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4290.47ms β
β Total Time (spill-dedup-view-arrays) β 4266.93ms β
β Average Time (HEAD) β 429.05ms β
β Average Time (spill-dedup-view-arrays) β 426.69ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.1 MiB | 174.9 MiB | -0.1% |
| 2 | 220.4 MiB | 219.6 MiB | -0.3% |
| 3 | 518.9 MiB | 539.3 MiB | +3.9% |
| 4 | 265.2 MiB | 264.9 MiB | -0.1% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.7 MiB | 355.2 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 499.7 MiB | +0.4% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 524.7 MiB | 522.2 MiB | -0.5% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`ed43e69 (merge-base)`) | 524.7 MiB | 1.2 GiB | 746.5 MiB | 2.4Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 539.3 MiB | 1.1 GiB | 602.7 MiB | 2.1Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 838.2 MiB |
| CPU user | 63.2s |
| CPU sys | 15.1s |
| Peak spill | 973.0 MiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 792.2 MiB |
| CPU user | 64.0s |
| CPU sys | 15.0s |
| Peak spill | 1013.6 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r3755018057
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -420,6 +420,20 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Garbage collect and deduplicate a string view before writing it out to disk
+///
+/// This is to avoid inflating disk usage and also to ensure that deduplication reduces memory pressure when reading back.
+fn gc_dedup_view<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> GenericByteViewArray<T> {
+ let mut builder = GenericByteViewBuilder::<T>::with_capacity(array.len())
Review Comment:
Yes this is the trade off. However, I believe the size of this is actually dictated by the configured batch size so it's never going to be more than that.
So with `16384` as the default batch size this could use ~ `16384 * 64 = 1MiB <+ hash internals>` RAM to write this out + CPU overhead of hash lookups which I think is ~ `log(n)`. So yeah high cardinality string views will probably cause increased memory/cpu but not by much. Keeping in mind that we are spilling here to reduce RAM usage, and so when we re-hydrate the spilled file, the memory usage is *much* more reduced for low cardinality views.
In high cardinality scenarios this will add overhead, but it's hard to measure without doing a second pass to work out what the cardinality is. Maybe there is a better way to gauge the cardinality of the input view and use that to decide? Not sure there is a good method for this.
But, in production we have seen a good reduction in file usage/memory pressure with gc & dedup (i.e, we have applied this as a patch to our version of DF), so I feel that for real use cases the trade off is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5779637711
## Summary: behavior under default settings and under memory pressure
### Default settings (no memory limit)
This PR has no effect. The changed code runs only from `InProgressSpillFile::append_batch` / `append_batch_async`, and operators call these only when a memory reservation fails. The default memory pool is unbounded, so no operator spills.
The three default-setting runs (tpch, tpcds, clickbench_partitioned) all report `Peak spill: 0 B`. Their per-query differences are noise: TPC-H shows "13 faster" and TPC-DS / ClickBench show "5 slower" for a code path that did not execute.
### Under memory pressure
On each spilled batch, for each `Utf8View` / `BinaryView` column (also when nested in `List`, `Map`, `Dictionary`, ...) with more than 10 KB of data buffers:
- **On `main`:** `gc()` copies the bytes of each non-inline view into new buffers. When views share bytes, `gc()` writes one copy per row. Parquet dictionary pages decode to views that all point into one dictionary buffer, so this is the normal case for low-cardinality string columns read from Parquet. This is the regression from https://github.com/apache/datafusion/pull/21633 that https://github.com/apache/datafusion/issues/23564 shows (1.3 MB Parquet file β 83.2 MB spilled).
- **With this PR:** `GenericByteViewBuilder::with_deduplicate_strings()` hashes each non-inline value (> 12 bytes) and writes each distinct value one time.
Consequences:
- **Disk and read-back memory:** smaller when a batch has repeated non-inline values. The dedup scope is one batch (default `batch_size` = 8192 rows), not the full spill file.
- **CPU:** one hash and one probe per non-inline value, on every spilled batch. When the values in a batch are unique, this cost gives no benefit.
- **Transient memory:** the hash table and the builder blocks. The memory pool does not track this memory, but it is small (hundreds of KB per 8192-row batch). The builder blocks have `capacity > len`, which is why the test now measures `len`. This does not change the IPC bytes that are written to disk.
### Measured results (sort_tpch SF1, `c4a-highmem-16`)
| Config | Result |
| --- | --- |
| 256 MiB, 12 partitions | Q3, Q7βQ11 fail on **both** sides (pre-existing). Other queries: no change. |
| 512 MiB, 12 partitions | Q7, Q10 fail on both sides. Q3 +4% (586.9 β 611.3 ms, low stddev). Others within Β±2%. |
| 512 MiB, 4 partitions | Q3 **+15%** (613.9 β 706.2 ms), Q11 +6%, Q8 +5%, Q9 +4%, Q10 +3%. Total +4%, CPU user +4%. |
The slower queries all include `l_comment` (4.5M distinct values, longer than 12 bytes). This is the worst case for this PR: each value is hashed and no value is deduplicated. This agrees with the concern in the review thread about unique strings.
Q1, Q2, Q4βQ6 have no non-inline strings, and they show no change, as expected. Q7 is the only query with a repeated non-inline column (`l_shipinstruct`: 4 distinct values, 2 of them > 12 bytes). It shows +1%, but the bot does not report spill bytes, so we do not know the disk effect.
Memory pool peaks do not change (Β±2%). This is also expected, because sort_tpch has no repeated long strings in its hot columns.
## Benchmarks that are missing
1. **The target case, with spill bytes measured.** No run measures the claim of this PR. The bot reports `Peak spill: 0 B` also for the sort_tpch runs that spilled, so we have no disk numbers. I recommend a main-vs-PR sweep that uses the #23564 repro and records `spilled_bytes`, `spill_count`, wall time, pool peak and RSS from `EXPLAIN ANALYZE`:
- distinct values per batch: 1, 100, 1000, 8192 (unique)
- string length: 16, 64, 256 bytes
- source: dictionary-encoded Parquet (views share bytes) and plain-encoded or computed strings (views do not share bytes)
- memory limit: a light spill (2β3 spill files) and a heavy spill
This gives the break-even cardinality, and it shows the size of the win.
2. **Real data.** Sort ClickBench `hits` under a memory limit. The `data_sort_pushdown` step in `bench.sh` already does `COPY (SELECT * FROM hits ORDER BY "EventTime")` with `memory_limit` set. `hits` has a realistic mix of low- and high-cardinality long strings. Measure wall time and `spilled_bytes`.
3. **Criterion microbenchmark** of `gc()` against `gc_dedup_view` on one batch, for the same cardinality and length matrix. For example, extend `datafusion/physical-plan/benches/spill_io.rs`. This isolates the per-batch CPU cost with low noise.
4. **Other spill producers.** Aggregation spills write group keys, and the keys are distinct in each spilled batch. For these columns, dedup cannot make the output smaller, so the hash cost is always overhead. `external_aggr` uses only integer keys, so it cannot show this. A string-key external aggregation (for example `SELECT l_comment, count(*) FROM lineitem GROUP BY l_comment` with a memory limit) is necessary. Repartition (`spill_pool`) and sort-merge join spills also use this path.
5. **A/A control** for the 512 MiB / 4 partitions sort_tpch run, to confirm that Q3 +15% is real.
## Possible mitigation (for discussion)
A cheap check can select between `gc()` and dedup without hashing the strings. For example, compare the sum of the non-inline view lengths with the total size of the data buffers. If the sum is larger, the views share bytes (as with Parquet dictionary pages), and `gc()` will make the array larger. Only then is dedup necessary. Another option is to preserve the sharing that exists: remap views by `(buffer_index, offset)` instead of by value. This does not hash the string bytes, and it keeps the output no larger than the input. Both options keep the #23564 fix and remove most of the cost on high-cardinality columns such as `l_comment`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783373942
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.33 ms β 1.31 ms β no change β
β QQuery 1 β 12.26 ms β 12.16 ms β no change β
β QQuery 2 β 37.53 ms β 37.12 ms β no change β
β QQuery 3 β 32.67 ms β 32.80 ms β no change β
β QQuery 4 β 250.82 ms β 260.40 ms β no change β
β QQuery 5 β 282.93 ms β 283.62 ms β no change β
β QQuery 6 β 1.37 ms β 1.37 ms β no change β
β QQuery 7 β 13.78 ms β 13.37 ms β no change β
β QQuery 8 β 359.68 ms β 349.36 ms β no change β
β QQuery 9 β 487.19 ms β 499.00 ms β no change β
β QQuery 10 β 65.14 ms β 66.42 ms β no change β
β QQuery 11 β 75.51 ms β 76.84 ms β no change β
β QQuery 12 β 275.54 ms β 275.26 ms β no change β
β QQuery 13 β 375.79 ms β 368.25 ms β no change β
β QQuery 14 β 291.06 ms β 288.18 ms β no change β
β QQuery 15 β 290.23 ms β 303.37 ms β no change β
β QQuery 16 β 630.20 ms β 671.74 ms β 1.07x slower β
β QQuery 17 β 627.33 ms β 652.79 ms β no change β
β QQuery 18 β 1358.28 ms β 1338.08 ms β no change β
β QQuery 19 β 28.42 ms β 28.10 ms β no change β
β QQuery 20 β 524.28 ms β 531.22 ms β no change β
β QQuery 21 β 521.70 ms β 527.61 ms β no change β
β QQuery 22 β 994.19 ms β 1004.34 ms β no change β
β QQuery 23 β 3149.94 ms β 3278.32 ms β no change β
β QQuery 24 β 41.06 ms β 42.75 ms β no change β
β QQuery 25 β 109.10 ms β 107.19 ms β no change β
β QQuery 26 β 42.64 ms β 41.36 ms β no change β
β QQuery 27 β 519.46 ms β 530.92 ms β no change β
β QQuery 28 β 2934.14 ms β 2918.47 ms β no change β
β QQuery 29 β 42.79 ms β 42.83 ms β no change β
β QQuery 30 β 328.53 ms β 319.58 ms β no change β
β QQuery 31 β 293.85 ms β 282.72 ms β no change β
β QQuery 32 β 1232.80 ms β 1137.19 ms β +1.08x faster β
β QQuery 33 β 1681.73 ms β 1629.23 ms β no change β
β QQuery 34 β 1740.73 ms β 1686.20 ms β no change β
β QQuery 35 β 340.51 ms β 333.75 ms β no change β
β QQuery 36 β 72.76 ms β 69.22 ms β no change β
β QQuery 37 β 37.12 ms β 37.77 ms β no change β
β QQuery 38 β 42.67 ms β 41.90 ms β no change β
β QQuery 39 β 148.15 ms β 156.86 ms β 1.06x slower β
β QQuery 40 β 15.28 ms β 16.47 ms β 1.08x slower β
β QQuery 41 β 14.88 ms β 15.79 ms β 1.06x slower β
β QQuery 42 β 14.49 ms β 14.72 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 20339.87ms β
β Total Time (spill-dedup-view-arrays) β 20325.96ms β
β Average Time (HEAD) β 473.02ms β
β Average Time (spill-dedup-view-arrays) β 472.70ms β
β Queries Faster β 1 β
β Queries Slower β 4 β
β Queries with No Change β 38 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.33 / 4.31 Β±5.81 / 15.93 ms β 1.31 / 4.33 Β±5.81 / 15.95 ms β no change β
β QQuery 1 β 12.26 / 12.46 Β±0.15 / 12.62 ms β 12.16 / 12.37 Β±0.13 / 12.56 ms β no change β
β QQuery 2 β 37.53 / 37.93 Β±0.40 / 38.48 ms β 37.12 / 37.52 Β±0.32 / 38.03 ms β no change β
β QQuery 3 β 32.67 / 33.36 Β±0.88 / 34.98 ms β 32.80 / 33.23 Β±0.48 / 34.10 ms β no change β
β QQuery 4 β 250.82 / 257.52 Β±4.32 / 262.29 ms β 260.40 / 265.11 Β±2.53 / 267.65 ms β no change β
β QQuery 5 β 282.93 / 295.18 Β±7.19 / 302.14 ms β 283.62 / 290.21 Β±4.44 / 297.33 ms β no change β
β QQuery 6 β 1.37 / 1.52 Β±0.24 / 1.99 ms β 1.37 / 1.52 Β±0.24 / 1.99 ms β no change β
β QQuery 7 β 13.78 / 13.94 Β±0.09 / 14.04 ms β 13.37 / 13.60 Β±0.14 / 13.77 ms β no change β
β QQuery 8 β 359.68 / 364.11 Β±4.23 / 371.03 ms β 349.36 / 361.79 Β±10.08 / 377.71 ms β no change β
β QQuery 9 β 487.19 / 495.73 Β±6.07 / 501.84 ms β 499.00 / 511.68 Β±10.82 / 530.73 ms β no change β
β QQuery 10 β 65.14 / 65.91 Β±0.95 / 67.72 ms β 66.42 / 67.19 Β±0.55 / 68.04 ms β no change β
β QQuery 11 β 75.51 / 75.88 Β±0.26 / 76.24 ms β 76.84 / 77.59 Β±0.58 / 78.26 ms β no change β
β QQuery 12 β 275.54 / 288.55 Β±8.07 / 299.96 ms β 275.26 / 280.01 Β±2.89 / 283.36 ms β no change β
β QQuery 13 β 375.79 / 404.15 Β±18.07 / 423.45 ms β 368.25 / 405.16 Β±21.29 / 432.98 ms β no change β
β QQuery 14 β 291.06 / 296.39 Β±4.70 / 304.86 ms β 288.18 / 294.01 Β±6.77 / 307.01 ms β no change β
β QQuery 15 β 290.23 / 308.59 Β±13.91 / 329.17 ms β 303.37 / 348.29 Β±42.35 / 404.22 ms β 1.13x slower β
β QQuery 16 β 630.20 / 636.96 Β±5.09 / 643.71 ms β 671.74 / 711.37 Β±53.00 / 810.09 ms β 1.12x slower β
β QQuery 17 β 627.33 / 663.17 Β±20.05 / 682.05 ms β 652.79 / 668.71 Β±11.31 / 683.69 ms β no change β
β QQuery 18 β 1358.28 / 1382.70 Β±22.86 / 1425.59 ms β 1338.08 / 1361.43 Β±16.69 / 1387.12 ms β no change β
β QQuery 19 β 28.42 / 31.48 Β±5.68 / 42.84 ms β 28.10 / 30.86 Β±5.12 / 41.11 ms β no change β
β QQuery 20 β 524.28 / 532.82 Β±10.85 / 554.09 ms β 531.22 / 536.37 Β±3.58 / 541.27 ms β no change β
β QQuery 21 β 521.70 / 525.84 Β±2.48 / 529.02 ms β 527.61 / 534.93 Β±7.38 / 547.63 ms β no change β
β QQuery 22 β 994.19 / 1046.04 Β±41.86 / 1121.77 ms β 1004.34 / 1047.41 Β±26.05 / 1070.46 ms β no change β
β QQuery 23 β 3149.94 / 3365.16 Β±114.92 / 3458.98 ms β 3278.32 / 3373.65 Β±77.08 / 3501.29 ms β no change β
β QQuery 24 β 41.06 / 159.12 Β±143.15 / 408.23 ms β 42.75 / 146.01 Β±129.45 / 394.92 ms β +1.09x faster β
β QQuery 25 β 109.10 / 110.82 Β±1.69 / 113.69 ms β 107.19 / 113.97 Β±9.52 / 132.85 ms β no change β
β QQuery 26 β 42.64 / 46.34 Β±4.96 / 56.14 ms β 41.36 / 42.14 Β±0.45 / 42.64 ms β +1.10x faster β
β QQuery 27 β 519.46 / 533.15 Β±11.96 / 555.07 ms β 530.92 / 546.19 Β±15.24 / 574.16 ms β no change β
β QQuery 28 β 2934.14 / 2988.23 Β±31.16 / 3029.91 ms β 2918.47 / 2965.01 Β±34.97 / 3018.45 ms β no change β
β QQuery 29 β 42.79 / 44.76 Β±3.58 / 51.91 ms β 42.83 / 52.25 Β±13.48 / 77.65 ms β 1.17x slower β
β QQuery 30 β 328.53 / 335.99 Β±10.16 / 356.15 ms β 319.58 / 328.42 Β±7.58 / 338.73 ms β no change β
β QQuery 31 β 293.85 / 311.86 Β±10.40 / 322.96 ms β 282.72 / 300.59 Β±16.99 / 329.79 ms β no change β
β QQuery 32 β 1232.80 / 1406.24 Β±128.47 / 1544.60 ms β 1137.19 / 1274.01 Β±107.09 / 1434.88 ms β +1.10x faster β
β QQuery 33 β 1681.73 / 1848.63 Β±228.53 / 2298.44 ms β 1629.23 / 1758.58 Β±82.98 / 1873.76 ms β no change β
β QQuery 34 β 1740.73 / 1781.03 Β±26.29 / 1810.84 ms β 1686.20 / 1750.26 Β±36.58 / 1796.13 ms β no change β
β QQuery 35 β 340.51 / 447.13 Β±165.35 / 775.78 ms β 333.75 / 413.50 Β±78.58 / 521.00 ms β +1.08x faster β
β QQuery 36 β 72.76 / 77.30 Β±4.35 / 83.60 ms β 69.22 / 74.65 Β±6.20 / 84.66 ms β no change β
β QQuery 37 β 37.12 / 39.16 Β±2.86 / 44.77 ms β 37.77 / 44.11 Β±4.89 / 49.08 ms β 1.13x slower β
β QQuery 38 β 42.67 / 44.23 Β±1.76 / 46.80 ms β 41.90 / 47.71 Β±5.60 / 58.17 ms β 1.08x slower β
β QQuery 39 β 148.15 / 172.43 Β±20.26 / 208.47 ms β 156.86 / 171.50 Β±9.38 / 185.05 ms β no change β
β QQuery 40 β 15.28 / 15.71 Β±0.60 / 16.86 ms β 16.47 / 17.75 Β±1.07 / 19.35 ms β 1.13x slower β
β QQuery 41 β 14.88 / 15.09 Β±0.22 / 15.51 ms β 15.79 / 16.44 Β±0.38 / 16.93 ms β 1.09x slower β
β QQuery 42 β 14.49 / 18.78 Β±7.97 / 34.72 ms β 14.72 / 15.00 Β±0.23 / 15.39 ms β +1.25x faster β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 21535.72ms β
β Total Time (spill-dedup-view-arrays) β 21346.41ms β
β Average Time (HEAD) β 500.83ms β
β Average Time (spill-dedup-view-arrays) β 496.43ms β
β Queries Faster β 5 β
β Queries Slower β 7 β
β Queries with No Change β 31 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 18.4 GiB |
| Avg memory | 6.1 GiB |
| CPU user | 1043.0s |
| CPU sys | 107.7s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 18.2 GiB |
| Avg memory | 6.0 GiB |
| CPU user | 1037.1s |
| CPU sys | 111.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783287413
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783076971)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.10 ms β 40.92 ms β no change β
β QQuery 2 β 18.93 ms β 18.90 ms β no change β
β QQuery 3 β 28.77 ms β 28.73 ms β no change β
β QQuery 4 β 17.51 ms β 17.35 ms β no change β
β QQuery 5 β 35.56 ms β 35.43 ms β no change β
β QQuery 6 β 16.43 ms β 16.22 ms β no change β
β QQuery 7 β 41.13 ms β 40.31 ms β no change β
β QQuery 8 β 40.31 ms β 40.35 ms β no change β
β QQuery 9 β 49.45 ms β 49.24 ms β no change β
β QQuery 10 β 40.61 ms β 40.63 ms β no change β
β QQuery 11 β 13.63 ms β 13.46 ms β no change β
β QQuery 12 β 21.32 ms β 20.94 ms β no change β
β QQuery 13 β 41.06 ms β 39.61 ms β no change β
β QQuery 14 β 25.59 ms β 24.89 ms β no change β
β QQuery 15 β 32.54 ms β 30.70 ms β +1.06x faster β
β QQuery 16 β 14.71 ms β 14.04 ms β no change β
β QQuery 17 β 77.20 ms β 72.58 ms β +1.06x faster β
β QQuery 18 β 63.39 ms β 61.00 ms β no change β
β QQuery 19 β 32.45 ms β 31.89 ms β no change β
β QQuery 20 β 33.15 ms β 31.75 ms β no change β
β QQuery 21 β 58.31 ms β 55.56 ms β no change β
β QQuery 22 β 14.48 ms β 13.83 ms β no change β
βββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 757.64ms β
β Total Time (spill-dedup-view-arrays) β 738.33ms β
β Average Time (HEAD) β 34.44ms β
β Average Time (spill-dedup-view-arrays) β 33.56ms β
β Queries Faster β 2 β
β Queries Slower β 0 β
β Queries with No Change β 20 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.10 / 42.03 Β±1.03 / 43.58 ms β 40.92 / 42.43 Β±1.12 / 43.93 ms β no change β
β QQuery 2 β 18.93 / 19.28 Β±0.32 / 19.84 ms β 18.90 / 18.99 Β±0.08 / 19.10 ms β no change β
β QQuery 3 β 28.77 / 29.03 Β±0.18 / 29.30 ms β 28.73 / 28.96 Β±0.15 / 29.18 ms β no change β
β QQuery 4 β 17.51 / 17.66 Β±0.09 / 17.77 ms β 17.35 / 17.48 Β±0.09 / 17.62 ms β no change β
β QQuery 5 β 35.56 / 36.60 Β±0.73 / 37.65 ms β 35.43 / 35.55 Β±0.09 / 35.69 ms β no change β
β QQuery 6 β 16.43 / 16.61 Β±0.15 / 16.79 ms β 16.22 / 16.93 Β±1.00 / 18.90 ms β no change β
β QQuery 7 β 41.13 / 42.39 Β±1.09 / 44.39 ms β 40.31 / 42.90 Β±1.52 / 44.57 ms β no change β
β QQuery 8 β 40.31 / 40.87 Β±0.52 / 41.73 ms β 40.35 / 40.73 Β±0.31 / 41.14 ms β no change β
β QQuery 9 β 49.45 / 50.17 Β±0.76 / 51.37 ms β 49.24 / 51.18 Β±1.10 / 52.48 ms β no change β
β QQuery 10 β 40.61 / 41.36 Β±1.10 / 43.52 ms β 40.63 / 41.25 Β±0.57 / 42.28 ms β no change β
β QQuery 11 β 13.63 / 13.81 Β±0.17 / 14.13 ms β 13.46 / 13.88 Β±0.41 / 14.66 ms β no change β
β QQuery 12 β 21.32 / 21.70 Β±0.52 / 22.71 ms β 20.94 / 21.16 Β±0.13 / 21.29 ms β no change β
β QQuery 13 β 41.06 / 41.69 Β±0.54 / 42.55 ms β 39.61 / 39.81 Β±0.14 / 39.98 ms β no change β
β QQuery 14 β 25.59 / 25.76 Β±0.09 / 25.87 ms β 24.89 / 25.70 Β±0.97 / 27.55 ms β no change β
β QQuery 15 β 32.54 / 32.97 Β±0.48 / 33.81 ms β 30.70 / 31.15 Β±0.45 / 31.96 ms β +1.06x faster β
β QQuery 16 β 14.71 / 15.07 Β±0.35 / 15.64 ms β 14.04 / 14.33 Β±0.15 / 14.45 ms β no change β
β QQuery 17 β 77.20 / 80.15 Β±2.00 / 83.32 ms β 72.58 / 73.31 Β±0.49 / 73.85 ms β +1.09x faster β
β QQuery 18 β 63.39 / 64.33 Β±0.96 / 66.01 ms β 61.00 / 62.50 Β±1.03 / 63.88 ms β no change β
β QQuery 19 β 32.45 / 33.05 Β±0.70 / 34.42 ms β 31.89 / 32.05 Β±0.12 / 32.26 ms β no change β
β QQuery 20 β 33.15 / 34.07 Β±0.86 / 35.41 ms β 31.75 / 32.13 Β±0.26 / 32.41 ms β +1.06x faster β
β QQuery 21 β 58.31 / 59.00 Β±0.78 / 60.53 ms β 55.56 / 57.54 Β±1.41 / 59.52 ms β no change β
β QQuery 22 β 14.48 / 14.77 Β±0.20 / 15.02 ms β 13.83 / 14.14 Β±0.22 / 14.42 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 772.38ms β
β Total Time (spill-dedup-view-arrays) β 754.09ms β
β Average Time (HEAD) β 35.11ms β
β Average Time (spill-dedup-view-arrays) β 34.28ms β
β Queries Faster β 3 β
β Queries Slower β 0 β
β Queries with No Change β 19 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 522.2 MiB |
| CPU user | 21.9s |
| CPU sys | 1.8s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.4 GiB |
| Avg memory | 518.3 MiB |
| CPU user | 21.1s |
| CPU sys | 1.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783901920
run benchmark spill_views
env:
QUERY: "5"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5785097696
## Benchmark bot results (head `cbdecca5ea`, merge-base `5b5bed9`)
A/B = main vs this PR. A/A = main vs main (same commit both sides), the noise control. All numbers are the bot's mean times; ratios are branch / base, so >1.00 is slower with this PR.
### 1. Default settings (no memory limit, nothing spills)
| Benchmark | A/B run 1 | A/B run 2 | A/A run 1 | A/A run 2 |
| --- | --- | --- | --- | --- |
| tpch_sf1 total | 0.99x | 0.99x | 0.97x | 1.00x |
| tpcds_sf1 total | 0.99x | 0.99x | 0.99x | 0.99x |
| clickbench_partitioned total | 1.02x | 1.01x | 1.00x | 1.02x |
No effect, as expected: without a memory limit no operator spills, so the changed code never runs. Every A/B value is inside the A/A range.
### 2. Memory pressure: sort_tpch SF1, 512M limit, 4 partitions
Mean ms, base β branch. Q1βQ7 and Q10 show no change in all four runs; Q3 fails on both sides.
| Query | A/B run 1 | A/B run 2 | A/A run 1 | A/A run 2 |
| --- | --- | --- | --- | --- |
| Q8 (`l_comment`) | 501 β 522 (1.04x) | 486 β 513 (1.06x) | 487 β 492 | 493 β 495 |
| Q9 (`l_comment`) | 556 β 573 (1.03x) | 536 β 565 (1.05x) | 530 β 535 | 535 β 548 |
| Q11 (`l_comment`) | 385 β 427 (1.11x) | 378 β 410 (1.08x) | 376 β 377 | 396 β 379 |
| Total | 4409 β 4477 (1.02x) | 4294 β 4398 (1.02x) | 1.00x | 1.00x |
### 3. spill_views (each query sets its own limit)
| Query | A/B | A/B run 2 | A/A |
| --- | --- | --- | --- |
| q01 sort, 1 distinct value | 106.8 β 90.8 ms (**0.85x**) | | 1.00x, 1.02x, 1.00x |
| q02 sort, 1000 distinct values | 102.4 β 100.3 ms (0.98x) | | 1.00x |
| q03 sort, BinaryView, 1000 distinct | 102.0 β 100.5 ms (0.99x) | | 1.00x |
| q04 sort, all distinct | 137.7 β 172.5 ms (**1.25x**) | 123.5 β 178.6 ms (**1.45x**) | 0.94x |
| q05 GROUP BY, all distinct | OOM on the branch side | OOM on the branch side | passes, 1.00x (3 runs) |
### 4. Spill size and memory (spill_views)
| Metric | A/B base | A/B branch | A/A base | A/A branch |
| --- | --- | --- | --- | --- |
| Peak spill, q01βq03 | 84.2 MiB | **30.6 MiB** | 84.2 MiB | 84.2 MiB |
| Peak spill, q04 | 84.2 MiB | 74.0 MiB | 52.4 MiB | 84.2 MiB |
| Peak RSS, q01βq03 | 412.6 MiB | 493.9 MiB | 432.4 MiB | 445.5 MiB |
Peak spill for q01βq03 is the point of the PR: 2.8x less written. The q04 and sort_tpch spill figures are sampled once per second and move by that much between identical runs (see the A/A columns), so only the q01βq03 figures are solid.
### 5. q05 out of memory
`spill_views` q05 (GROUP BY on an all-distinct string key, 96M limit, greedy pool) fails on the branch side with `ResourcesExhausted` in `FinalHashAggregateStream`. The whole suite run aborts when it happens.
| | Runs | Result |
| --- | --- | --- |
| A/B (main vs PR) | 4 | 4 failed, always on the branch side |
| A/A (main vs main) | 3 | 0 failed |
Locally on macOS the same query at 96M fails about 1 in 12 runs on **both** builds, and at 160M it fails on both (10/12 main, 12/12 PR), so the limit interacts with spilling in a way that predates this PR. But on the bot the split is clean, and peak RSS for q01βq03 is about 80 MiB higher on the branch side, which suggests the PR does need more headroom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789892426
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789851638)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789851638-2630-tfw7r 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 7570366fd929daf9ced744bb8397686b50565b18 (7570366fd929daf9ced744bb8397686b50565b18) to 7570366fd929daf9ced744bb8397686b50565b18 [diff](https://github.com/apache/datafusion/compare/7570366fd929daf9ced744bb8397686b50565b18..7570366fd929daf9ced744bb8397686b50565b18)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
baseline:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
changed:
ref: "7570366fd929daf9ced744bb8397686b50565b18"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784334244
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784306558)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5784306558-2616-pdmrs 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784305266
run benchmark spill_views
env:
BENCH_QUERY: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783755273
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783724486-2599-2b86g 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784141492
Benchmark for [this request](https://github.com/apache/datafusion/pull/23565#issuecomment-5783901920) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `spill_views`
<details><summary>Runner log (last 40 lines)</summary>
```
Compiling datafusion-datasource-json v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-json)
Compiling datafusion-datasource-csv v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-csv)
Compiling datafusion-datasource-arrow v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-arrow)
Compiling datafusion-datasource-parquet v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-parquet)
Compiling datafusion-physical-optimizer v55.1.0 (/workspace/datafusion-branch/datafusion/physical-optimizer)
Compiling datafusion-catalog-listing v55.1.0 (/workspace/datafusion-branch/datafusion/catalog-listing)
Compiling datafusion-functions-table v55.1.0 (/workspace/datafusion-branch/datafusion/functions-table)
Compiling datafusion v55.1.0 (/workspace/datafusion-branch/datafusion/core)
Compiling datafusion-proto v55.1.0 (/workspace/datafusion-branch/datafusion/proto)
Compiling datafusion-benchmarks v55.1.0 (/workspace/datafusion-branch/benchmarks)
Finished `bench` profile [optimized] target(s) in 5m 17s
Running benches/sql.rs (/workspace/datafusion-branch/target/release/deps/sql-2af2193982da74c5)
Gnuplot not found, using plotters backend
Benchmarking spill_views/q01_sort_string_1_distinct_repeated
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Collecting 10 samples in estimated 5.3800 s (60 iterations)
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Analyzing
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Collecting 10 samples in estimated 5.9477 s (60 iterations)
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Collecting 10 samples in estimated 5.9114 s (60 iterations)
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q04_sort_string_all_distinct_distinct
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Collecting 10 samples in estimated 5.4377 s (30 iterations)
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Analyzing
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Warming up for 3.0000 s
thread 'main' (15079) panicked at benchmarks/src/sql_benchmark_runner.rs:224:25:
Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted("Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\n FinalHashAggregateStream[1]#5583(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[2]#5584(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[0]#5590(can spill: true) consumed 27.1 MB, peak 27.1 MB,\n RepartitionExec[0]#5589(can spill: true) consumed 4.8 MB, peak 4.8 MB,\n RepartitionExec[2]#5592(can spill: true) consumed 2.4 MB, peak 2.4 MB.\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' (15079) panicked at benchmarks/benches/sql.rs:55:27:
failed to run SQL benchmarks: Execution("criterion benchmark failed: Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted(\"Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\\n FinalHashAggregateStream[1]#5583(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[2]#5584(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[0]#5590(can spill: true) consumed 27.1 MB, peak 27.1 MB,\\n RepartitionExec[0]#5589(can spill: true) consumed 4.8 MB, peak 4.8 MB,\\n RepartitionExec[2]#5592(can spill: true) consumed 2.4 MB, peak 2.4 MB.\\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)\")")
error: bench failed, to rerun pass `--bench sql`
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784058859
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783904761)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783904761-2611-hjrnj 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
QUERY: "5"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5786504735
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5786320092)
**Instance:** `c4a-highmem-32` (16 vCPU / 128 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_SUBGROUP: "repeated"
resources:
cpu: "16"
memory: "128Gi"
arch: "arm64"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 32
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 2 MiB (32 instances)
L1i cache: 2 MiB (32 instances)
L2 cache: 64 MiB (32 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.18 104.4Β±0.95ms ? ?/sec 1.00 88.4Β±0.44ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.02 99.0Β±0.53ms ? ?/sec 1.00 97.5Β±0.49ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 97.7Β±0.48ms ? ?/sec 1.00 97.3Β±0.43ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 275.2s |
| Peak memory | 507.1 MiB |
| Avg memory | 43.7 MiB |
| CPU user | 31.2s |
| CPU sys | 18.2s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 275.2s |
| Peak memory | 555.0 MiB |
| Avg memory | 47.4 MiB |
| CPU user | 37.9s |
| CPU sys | 7.6s |
| Peak spill | 30.6 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789149386
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789124786)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789124786-2626-cgszq 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789126649
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863244409
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863112345)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863112345-2864-2kcn9 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5867439410
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5867397520)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5867397520-2876-4ptw6 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863305354
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863112345)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.62 ms β 134.92 ms β no change β
β Q2 β 136.55 ms β 135.39 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.40 ms β 199.12 ms β no change β
β Q5 β 230.83 ms β 226.79 ms β no change β
β Q6 β 253.10 ms β 248.67 ms β no change β
β Q7 β 691.56 ms β 682.69 ms β no change β
β Q8 β 478.50 ms β 484.68 ms β no change β
β Q9 β 536.59 ms β 529.86 ms β no change β
β Q10 β 1232.51 ms β 1250.78 ms β no change β
β Q11 β 375.39 ms β 374.12 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4269.05ms β
β Total Time (spill-dedup-view-arrays) β 4267.02ms β
β Average Time (HEAD) β 426.90ms β
β Average Time (spill-dedup-view-arrays) β 426.70ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 135.62 / 137.49 Β±1.05 / 138.44 ms β 134.92 / 136.04 Β±1.18 / 137.71 ms β no change β
β Q2 β 136.55 / 137.23 Β±0.61 / 138.21 ms β 135.39 / 137.08 Β±1.26 / 138.88 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 198.40 / 204.32 Β±4.40 / 210.84 ms β 199.12 / 202.14 Β±1.77 / 204.21 ms β no change β
β Q5 β 230.83 / 233.74 Β±1.67 / 235.95 ms β 226.79 / 231.26 Β±4.28 / 238.62 ms β no change β
β Q6 β 253.10 / 255.71 Β±2.39 / 259.42 ms β 248.67 / 252.86 Β±2.73 / 256.28 ms β no change β
β Q7 β 691.56 / 697.98 Β±7.42 / 712.27 ms β 682.69 / 694.83 Β±7.71 / 701.99 ms β no change β
β Q8 β 478.50 / 490.22 Β±7.50 / 500.33 ms β 484.68 / 488.94 Β±5.87 / 500.51 ms β no change β
β Q9 β 536.59 / 540.21 Β±2.72 / 544.80 ms β 529.86 / 541.52 Β±10.22 / 557.30 ms β no change β
β Q10 β 1232.51 / 1243.05 Β±7.36 / 1253.93 ms β 1250.78 / 1262.22 Β±7.44 / 1271.37 ms β no change β
β Q11 β 375.39 / 378.76 Β±3.54 / 384.34 ms β 374.12 / 378.91 Β±3.97 / 384.06 ms β no change β
βββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4318.70ms β
β Total Time (spill-dedup-view-arrays) β 4325.79ms β
β Average Time (HEAD) β 431.87ms β
β Average Time (spill-dedup-view-arrays) β 432.58ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 10 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.5 MiB | 174.2 MiB | -0.7% |
| 2 | 218.9 MiB | 220.4 MiB | +0.7% |
| 3 | 511.4 MiB | 552.2 MiB | +8.0% |
| 4 | 264.6 MiB | 265.2 MiB | +0.2% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.7 MiB | 355.2 MiB | -0.1% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 497.9 MiB | +0.0% |
| 10 | 510.1 MiB | 510.1 MiB | +0.0% |
| 11 | 522.9 MiB | 522.3 MiB | -0.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`ed43e69 (merge-base)`) | 522.9 MiB | 1.3 GiB | 776.7 MiB | 2.5Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 552.2 MiB | 1.2 GiB | 641.2 MiB | 2.2Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 829.8 MiB |
| CPU user | 62.9s |
| CPU sys | 15.6s |
| Peak spill | 1.1 GiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 867.9 MiB |
| CPU user | 66.2s |
| CPU sys | 16.1s |
| Peak spill | 1.1 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863476182
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863111696)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing ed43e69fdca8737c9d065b53b5fc5e4183302775 (ed43e69fdca8737c9d065b53b5fc5e4183302775) to ed43e69fdca8737c9d065b53b5fc5e4183302775 [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..ed43e69fdca8737c9d065b53b5fc5e4183302775)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
baseline:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
changed:
ref: "ed43e69fdca8737c9d065b53b5fc5e4183302775"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 106.1Β±1.55ms ? ?/sec 1.01 106.9Β±0.84ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 101.6Β±0.50ms ? ?/sec 1.01 102.7Β±0.77ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 100.1Β±0.58ms ? ?/sec 1.01 100.9Β±0.42ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 130.0Β±0.89ms ? ?/sec 1.01 131.6Β±1.24ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 112.1Β±1.08ms ? ?/sec 1.00 112.6Β±0.69ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69fdca8737c9d065b53b5fc5e4183302775` | Changed: `ed43e69fdca8737c9d065b53b5fc5e4183302775`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 138.8 MiB | 138.8 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 120.6 MiB | 120.6 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 138.8 MiB | 832.4 MiB | 693.7 MiB | 6.0Γ |
| spill_views | changed (`ed43e69fdca8737c9d065b53b5fc5e4183302775`) | 138.8 MiB | 818.6 MiB | 679.8 MiB | 5.9Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 375.2s |
| Peak memory | 832.4 MiB |
| Avg memory | 56.6 MiB |
| CPU user | 66.6s |
| CPU sys | 29.3s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 615.3s |
| Peak memory | 818.6 MiB |
| Avg memory | 34.4 MiB |
| CPU user | 64.7s |
| CPU sys | 28.6s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969067092
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4969050947-1047-9mz5q 6.12.85+ #1 SMP Mon May 11 08:17:35 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (25cc81b71537b63a872d994c8b0512b0bb715fa0) to f755cb4 (merge-base) [diff](https://github.com/apache/datafusion/compare/f755cb44fa33c69837a8774be862fbfe01e7a100..25cc81b71537b63a872d994c8b0512b0bb715fa0) using: clickbench_partitioned
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-4969237586
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-4969050947)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.25 / 4.02 Β±5.44 / 14.89 ms β 1.22 / 3.99 Β±5.44 / 14.88 ms β no change β
β QQuery 1 β 12.70 / 12.84 Β±0.09 / 12.95 ms β 12.40 / 12.77 Β±0.27 / 13.23 ms β no change β
β QQuery 2 β 35.80 / 36.36 Β±0.36 / 36.79 ms β 35.67 / 35.95 Β±0.36 / 36.61 ms β no change β
β QQuery 3 β 30.82 / 31.49 Β±0.68 / 32.75 ms β 30.59 / 30.80 Β±0.19 / 31.15 ms β no change β
β QQuery 4 β 219.80 / 228.85 Β±4.65 / 232.90 ms β 220.22 / 227.50 Β±4.02 / 230.88 ms β no change β
β QQuery 5 β 272.53 / 275.50 Β±2.40 / 279.69 ms β 268.71 / 274.43 Β±4.67 / 282.29 ms β no change β
β QQuery 6 β 1.26 / 1.41 Β±0.24 / 1.88 ms β 1.24 / 1.41 Β±0.24 / 1.88 ms β no change β
β QQuery 7 β 13.53 / 13.67 Β±0.16 / 13.91 ms β 13.40 / 13.61 Β±0.15 / 13.78 ms β no change β
β QQuery 8 β 318.06 / 326.57 Β±5.60 / 332.74 ms β 323.03 / 331.22 Β±4.58 / 336.78 ms β no change β
β QQuery 9 β 457.93 / 463.57 Β±3.65 / 468.97 ms β 455.97 / 468.33 Β±10.04 / 484.29 ms β no change β
β QQuery 10 β 70.47 / 71.30 Β±0.73 / 72.40 ms β 70.60 / 75.63 Β±8.38 / 92.35 ms β 1.06x slower β
β QQuery 11 β 81.92 / 82.76 Β±0.50 / 83.44 ms β 80.64 / 82.29 Β±2.61 / 87.47 ms β no change β
β QQuery 12 β 271.52 / 276.53 Β±4.62 / 284.48 ms β 268.29 / 279.93 Β±8.78 / 294.33 ms β no change β
β QQuery 13 β 378.37 / 393.95 Β±14.04 / 418.05 ms β 375.01 / 383.14 Β±10.56 / 403.19 ms β no change β
β QQuery 14 β 285.60 / 291.61 Β±4.83 / 299.89 ms β 279.54 / 292.76 Β±7.92 / 300.60 ms β no change β
β QQuery 15 β 273.96 / 280.09 Β±3.87 / 284.58 ms β 274.98 / 285.61 Β±7.05 / 292.92 ms β no change β
β QQuery 16 β 627.08 / 639.96 Β±12.19 / 656.35 ms β 624.06 / 634.64 Β±15.43 / 665.17 ms β no change β
β QQuery 17 β 628.06 / 637.35 Β±12.65 / 661.47 ms β 627.76 / 636.06 Β±5.87 / 643.34 ms β no change β
β QQuery 18 β 1277.71 / 1297.36 Β±16.85 / 1326.15 ms β 1273.05 / 1309.98 Β±25.78 / 1349.45 ms β no change β
β QQuery 19 β 28.87 / 34.42 Β±9.45 / 53.27 ms β 27.47 / 27.74 Β±0.21 / 28.02 ms β +1.24x faster β
β QQuery 20 β 515.66 / 527.27 Β±11.49 / 543.78 ms β 519.96 / 525.18 Β±5.83 / 536.00 ms β no change β
β QQuery 21 β 515.86 / 522.39 Β±6.74 / 534.31 ms β 514.76 / 523.97 Β±8.11 / 537.60 ms β no change β
β QQuery 22 β 993.42 / 999.55 Β±6.77 / 1009.95 ms β 981.34 / 996.05 Β±13.43 / 1014.97 ms β no change β
β QQuery 23 β 3062.01 / 3098.23 Β±38.93 / 3173.76 ms β 3068.63 / 3120.26 Β±37.76 / 3162.40 ms β no change β
β QQuery 24 β 41.41 / 42.73 Β±1.37 / 45.20 ms β 40.81 / 42.03 Β±0.89 / 43.45 ms β no change β
β QQuery 25 β 113.66 / 114.04 Β±0.41 / 114.76 ms β 111.03 / 111.93 Β±0.78 / 113.33 ms β no change β
β QQuery 26 β 41.52 / 42.20 Β±0.47 / 43.00 ms β 41.58 / 43.34 Β±2.05 / 46.74 ms β no change β
β QQuery 27 β 670.59 / 679.48 Β±10.59 / 700.12 ms β 668.73 / 681.20 Β±7.41 / 690.93 ms β no change β
β QQuery 28 β 3019.28 / 3046.35 Β±23.56 / 3084.36 ms β 3060.72 / 3075.89 Β±18.86 / 3112.02 ms β no change β
β QQuery 29 β 40.87 / 42.24 Β±2.20 / 46.63 ms β 40.88 / 48.06 Β±12.64 / 73.28 ms β 1.14x slower β
β QQuery 30 β 307.57 / 318.44 Β±18.14 / 354.52 ms β 299.62 / 307.09 Β±7.17 / 319.99 ms β no change β
β QQuery 31 β 278.20 / 290.83 Β±7.95 / 301.96 ms β 277.20 / 293.43 Β±12.97 / 309.85 ms β no change β
β QQuery 32 β 923.31 / 957.06 Β±30.85 / 1003.81 ms β 949.60 / 971.42 Β±17.08 / 990.64 ms β no change β
β QQuery 33 β 1477.83 / 1499.48 Β±15.84 / 1515.48 ms β 1434.93 / 1489.19 Β±35.53 / 1530.22 ms β no change β
β QQuery 34 β 1497.68 / 1529.92 Β±17.91 / 1546.25 ms β 1491.65 / 1528.99 Β±25.64 / 1561.06 ms β no change β
β QQuery 35 β 278.59 / 306.51 Β±29.30 / 358.79 ms β 283.22 / 335.27 Β±74.30 / 482.55 ms β 1.09x slower β
β QQuery 36 β 66.18 / 72.47 Β±4.66 / 78.49 ms β 65.94 / 77.11 Β±10.67 / 96.44 ms β 1.06x slower β
β QQuery 37 β 35.48 / 37.67 Β±2.20 / 41.91 ms β 35.49 / 37.25 Β±1.92 / 40.79 ms β no change β
β QQuery 38 β 41.60 / 42.64 Β±1.12 / 44.74 ms β 41.43 / 48.82 Β±13.17 / 75.13 ms β 1.14x slower β
β QQuery 39 β 143.52 / 151.58 Β±8.57 / 167.87 ms β 138.54 / 151.49 Β±10.32 / 163.38 ms β no change β
β QQuery 40 β 14.10 / 14.89 Β±0.46 / 15.53 ms β 14.07 / 14.42 Β±0.34 / 15.05 ms β no change β
β QQuery 41 β 13.71 / 13.99 Β±0.32 / 14.58 ms β 13.55 / 14.05 Β±0.31 / 14.52 ms β no change β
β QQuery 42 β 13.11 / 13.49 Β±0.32 / 13.98 ms β 13.11 / 13.40 Β±0.17 / 13.65 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 19763.05ms β
β Total Time (spill-dedup-view-arrays) β 19857.62ms β
β Average Time (HEAD) β 459.61ms β
β Average Time (spill-dedup-view-arrays) β 461.81ms β
β Queries Faster β 1 β
β Queries Slower β 5 β
β Queries with No Change β 37 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 11.4 GiB |
| Avg memory | 4.6 GiB |
| CPU user | 1018.4s |
| CPU sys | 68.9s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 100.0s |
| Peak memory | 10.3 GiB |
| Avg memory | 4.1 GiB |
| CPU user | 1014.1s |
| CPU sys | 71.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "cetra3 (via GitHub)" <gi...@apache.org>.
cetra3 commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r3755018057
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -420,6 +420,20 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) -> Result<RecordBatch> {
}
}
+/// Garbage collect and deduplicate a string view before writing it out to disk
+///
+/// This is to avoid inflating disk usage and also to ensure that deduplication reduces memory pressure when reading back.
+fn gc_dedup_view<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> GenericByteViewArray<T> {
+ let mut builder = GenericByteViewBuilder::<T>::with_capacity(array.len())
Review Comment:
Yes this is the trade off. However, I believe the size of this is actually dictated by the configured batch size so it's never going to be more than that.
So with `16384` as the default batch size this could use ~ `16384 * 64 = 1MiB <+ hash internals>` RAM to write this out + CPU overhead of hash lookups which I think is ~ `log(n)`. So yeah high cardinality string views will probably cause increased memory/cpu but not by much. Keeping in mind that we are spilling here to reduce RAM usage, and so when we consider the re-hydrated spilled file, the memory usage is *much* more reduced for low cardinality views.
In high cardinality scenarios this will add overhead, but it's hard to measure without doing a second pass to work out what the cardinality is. Maybe there is a better way to gauge the cardinality of the input view and use that to decide? Not sure there is a good method for this.
But, in production we have seen a good reduction in file usage/memory pressure with gc & dedup (i.e, we have applied this as a patch to our version of DF), so I feel that for real use cases the trade off is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5249014582
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5249003089)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5249003089-1540-tzvxj 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248895426
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5248861843)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5248861843-1539-vqgpb 6.12.85+ #1 SMP Wed Jun 17 20:31:55 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5248953570
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5248861843)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (20bc3e1a140f8519ce80af05a8d446de9a505d8a) to a9b61ab (merge-base) [diff](https://github.com/apache/datafusion/compare/a9b61abf04b59ff08e7254b598b07f4fa058c5d1..20bc3e1a140f8519ce80af05a8d446de9a505d8a)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 120.96 ms β 121.04 ms β no change β
β Q2 β 105.45 ms β 104.87 ms β no change β
β Q3 β 582.82 ms β 609.50 ms β no change β
β Q4 β 197.22 ms β 196.43 ms β no change β
β Q5 β 264.57 ms β 263.53 ms β no change β
β Q6 β 276.51 ms β 276.76 ms β no change β
β Q7 β FAIL β FAIL β incomparable β
β Q8 β 382.54 ms β 387.57 ms β no change β
β Q9 β 413.23 ms β 415.20 ms β no change β
β Q10 β FAIL β FAIL β incomparable β
β Q11 β 247.95 ms β 252.52 ms β no change β
βββββββββ΄ββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 2591.24ms β
β Total Time (spill-dedup-view-arrays) β 2627.42ms β
β Average Time (HEAD) β 287.92ms β
β Average Time (spill-dedup-view-arrays) β 291.94ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 2 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 120.96 / 121.64 Β±0.87 / 123.23 ms β 121.04 / 121.94 Β±0.84 / 123.53 ms β no change β
β Q2 β 105.45 / 106.33 Β±1.05 / 107.90 ms β 104.87 / 106.37 Β±1.41 / 108.52 ms β no change β
β Q3 β 582.82 / 586.85 Β±3.16 / 591.54 ms β 609.50 / 611.27 Β±1.30 / 612.81 ms β no change β
β Q4 β 197.22 / 199.23 Β±2.16 / 202.18 ms β 196.43 / 198.91 Β±1.58 / 201.36 ms β no change β
β Q5 β 264.57 / 265.77 Β±1.12 / 267.35 ms β 263.53 / 264.73 Β±1.46 / 267.11 ms β no change β
β Q6 β 276.51 / 278.07 Β±1.64 / 281.14 ms β 276.76 / 277.97 Β±0.83 / 279.34 ms β no change β
β Q7 β FAIL β FAIL β incomparable β
β Q8 β 382.54 / 388.93 Β±4.68 / 395.35 ms β 387.57 / 396.01 Β±6.72 / 405.82 ms β no change β
β Q9 β 413.23 / 418.91 Β±8.37 / 435.57 ms β 415.20 / 421.61 Β±5.88 / 429.31 ms β no change β
β Q10 β FAIL β FAIL β incomparable β
β Q11 β 247.95 / 256.02 Β±7.97 / 270.72 ms β 252.52 / 259.40 Β±4.67 / 265.63 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 2621.75ms β
β Total Time (spill-dedup-view-arrays) β 2658.21ms β
β Average Time (HEAD) β 291.31ms β
β Average Time (spill-dedup-view-arrays) β 295.36ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 9 β
β Queries with Failure β 2 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `a9b61ab (merge-base)` | Changed: `spill-dedup-view-arrays`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 240.1 MiB | 241.2 MiB | +0.4% |
| 2 | 279.9 MiB | 282.1 MiB | +0.8% |
| 3 | 510.7 MiB | 503.0 MiB | -1.5% |
| 4 | 319.6 MiB | 322.1 MiB | +0.8% |
| 5 | 316.8 MiB | 321.1 MiB | +1.4% |
| 6 | 405.0 MiB | 407.1 MiB | +0.5% |
| 7 | 506.0 MiB | 502.8 MiB | -0.6% |
| 8 | 502.1 MiB | 503.2 MiB | +0.2% |
| 9 | 500.7 MiB | 498.1 MiB | -0.5% |
| 10 | 503.6 MiB | 509.8 MiB | +1.2% |
| 11 | 508.3 MiB | 520.5 MiB | +2.4% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`a9b61ab (merge-base)`) | 510.7 MiB | 1.7 GiB | 1.2 GiB | 3.4Γ |
| sort_tpch | changed (`spill-dedup-view-arrays`) | 520.5 MiB | 1.5 GiB | 1.0 GiB | 3.0Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 15.0s |
| Peak memory | 1.7 GiB |
| Avg memory | 923.5 MiB |
| CPU user | 52.0s |
| CPU sys | 7.8s |
| Peak spill | 0 B |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 15.0s |
| Peak memory | 1.5 GiB |
| Avg memory | 882.8 MiB |
| CPU user | 53.5s |
| CPU sys | 7.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5870709763
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5870585310)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (9c2f8f19f7767798cb02e9462a70f0f62ddb3059) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..9c2f8f19f7767798cb02e9462a70f0f62ddb3059)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.44 ms β 41.93 ms β no change β
β QQuery 2 β 19.08 ms β 19.47 ms β no change β
β QQuery 3 β 29.60 ms β 29.67 ms β no change β
β QQuery 4 β 18.06 ms β 18.00 ms β no change β
β QQuery 5 β 36.98 ms β 37.11 ms β no change β
β QQuery 6 β 16.94 ms β 17.00 ms β no change β
β QQuery 7 β 44.46 ms β 43.60 ms β no change β
β QQuery 8 β 42.24 ms β 42.70 ms β no change β
β QQuery 9 β 52.11 ms β 52.79 ms β no change β
β QQuery 10 β 42.62 ms β 42.64 ms β no change β
β QQuery 11 β 14.59 ms β 14.66 ms β no change β
β QQuery 12 β 22.16 ms β 22.33 ms β no change β
β QQuery 13 β 43.77 ms β 45.14 ms β no change β
β QQuery 14 β 25.50 ms β 26.01 ms β no change β
β QQuery 15 β 32.19 ms β 32.94 ms β no change β
β QQuery 16 β 14.50 ms β 14.82 ms β no change β
β QQuery 17 β 80.76 ms β 82.46 ms β no change β
β QQuery 18 β 65.86 ms β 67.84 ms β no change β
β QQuery 19 β 32.71 ms β 33.91 ms β no change β
β QQuery 20 β 32.97 ms β 33.49 ms β no change β
β QQuery 21 β 58.62 ms β 59.34 ms β no change β
β QQuery 22 β 14.86 ms β 15.04 ms β no change β
βββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 782.04ms β
β Total Time (spill-dedup-view-arrays) β 792.89ms β
β Average Time (HEAD) β 35.55ms β
β Average Time (spill-dedup-view-arrays) β 36.04ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 22 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.44 / 42.19 Β±1.06 / 44.27 ms β 41.93 / 43.00 Β±1.24 / 44.78 ms β no change β
β QQuery 2 β 19.08 / 19.50 Β±0.40 / 20.16 ms β 19.47 / 20.04 Β±0.59 / 21.18 ms β no change β
β QQuery 3 β 29.60 / 29.77 Β±0.11 / 29.94 ms β 29.67 / 30.08 Β±0.48 / 31.00 ms β no change β
β QQuery 4 β 18.06 / 18.66 Β±0.80 / 20.12 ms β 18.00 / 18.62 Β±0.85 / 20.27 ms β no change β
β QQuery 5 β 36.98 / 37.86 Β±0.63 / 38.77 ms β 37.11 / 37.37 Β±0.19 / 37.62 ms β no change β
β QQuery 6 β 16.94 / 17.21 Β±0.22 / 17.60 ms β 17.00 / 17.17 Β±0.10 / 17.31 ms β no change β
β QQuery 7 β 44.46 / 45.79 Β±0.84 / 46.99 ms β 43.60 / 45.24 Β±1.18 / 46.92 ms β no change β
β QQuery 8 β 42.24 / 43.02 Β±0.63 / 44.06 ms β 42.70 / 43.58 Β±0.76 / 44.85 ms β no change β
β QQuery 9 β 52.11 / 53.57 Β±1.07 / 55.22 ms β 52.79 / 54.47 Β±0.98 / 55.61 ms β no change β
β QQuery 10 β 42.62 / 44.01 Β±1.27 / 45.66 ms β 42.64 / 43.19 Β±0.37 / 43.78 ms β no change β
β QQuery 11 β 14.59 / 15.13 Β±0.76 / 16.63 ms β 14.66 / 14.86 Β±0.16 / 15.10 ms β no change β
β QQuery 12 β 22.16 / 22.53 Β±0.33 / 23.14 ms β 22.33 / 22.58 Β±0.18 / 22.87 ms β no change β
β QQuery 13 β 43.77 / 44.76 Β±0.76 / 46.08 ms β 45.14 / 46.82 Β±1.30 / 48.39 ms β no change β
β QQuery 14 β 25.50 / 26.81 Β±1.37 / 28.78 ms β 26.01 / 27.13 Β±1.58 / 30.21 ms β no change β
β QQuery 15 β 32.19 / 32.50 Β±0.37 / 33.23 ms β 32.94 / 34.32 Β±0.82 / 35.29 ms β 1.06x slower β
β QQuery 16 β 14.50 / 14.76 Β±0.28 / 15.25 ms β 14.82 / 15.11 Β±0.19 / 15.40 ms β no change β
β QQuery 17 β 80.76 / 82.76 Β±2.49 / 87.46 ms β 82.46 / 83.75 Β±1.24 / 85.82 ms β no change β
β QQuery 18 β 65.86 / 68.00 Β±1.62 / 70.39 ms β 67.84 / 69.83 Β±1.44 / 71.96 ms β no change β
β QQuery 19 β 32.71 / 33.73 Β±1.14 / 35.66 ms β 33.91 / 34.57 Β±0.76 / 35.92 ms β no change β
β QQuery 20 β 32.97 / 33.34 Β±0.26 / 33.64 ms β 33.49 / 34.09 Β±0.35 / 34.56 ms β no change β
β QQuery 21 β 58.62 / 60.07 Β±0.82 / 60.85 ms β 59.34 / 60.32 Β±0.76 / 61.63 ms β no change β
β QQuery 22 β 14.86 / 17.20 Β±4.11 / 25.41 ms β 15.04 / 16.28 Β±1.99 / 20.24 ms β +1.06x faster β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 803.17ms β
β Total Time (spill-dedup-view-arrays) β 812.41ms β
β Average Time (HEAD) β 36.51ms β
β Average Time (spill-dedup-view-arrays) β 36.93ms β
β Queries Faster β 1 β
β Queries Slower β 1 β
β Queries with No Change β 20 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`tpch` β `tpch_sf1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 1 | 21.4 MiB | 21.4 MiB | +0.0% |
| Query 2 | 15.7 MiB | 15.7 MiB | +0.0% |
| Query 3 | 11.5 MiB | 11.5 MiB | +0.0% |
| Query 4 | 23.0 MiB | 23.2 MiB | +0.7% |
| Query 5 | 52.6 MiB | 43.0 MiB | -18.1% |
| Query 6 | 832 B | 832 B | +0.0% |
| Query 7 | 107.4 MiB | 107.4 MiB | +0.0% |
| Query 8 | 33.8 MiB | 33.1 MiB | -2.1% |
| Query 9 | 153.9 MiB | 153.9 MiB | +0.0% |
| Query 10 | 36.7 MiB | 34.9 MiB | -5.0% |
| Query 11 | 72.0 MiB | 51.5 MiB | -28.5% |
| Query 12 | 20.0 MiB | 20.0 MiB | -0.0% |
| Query 13 | 52.3 MiB | 62.3 MiB | +19.1% |
| Query 14 | 7.7 MiB | 7.7 MiB | +0.0% |
| Query 15 | 12.2 MiB | 12.1 MiB | -0.8% |
| Query 16 | 44.3 MiB | 44.4 MiB | +0.2% |
| Query 17 | 142.2 MiB | 141.2 MiB | -0.7% |
| Query 18 | 358.3 MiB | 358.3 MiB | +0.0% |
| Query 19 | 746.1 KiB | 746.1 KiB | +0.0% |
| Query 20 | 61.7 MiB | 61.7 MiB | -0.1% |
| Query 21 | 41.7 MiB | 41.7 MiB | +0.0% |
| Query 22 | 32.4 MiB | 32.1 MiB | -0.9% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| tpch | base (`ed43e69 (merge-base)`) | 358.3 MiB | 1.3 GiB | 998.5 MiB | 3.8Γ |
| tpch | changed (`spill-dedup-view-arrays`) | 358.3 MiB | 1.1 GiB | 796.7 MiB | 3.2Γ |
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 717.5 MiB |
| CPU user | 22.7s |
| CPU sys | 1.9s |
| Peak spill | 3.3 MiB |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 647.4 MiB |
| CPU user | 22.7s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863135361
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110591)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5863110591-2860-sqp2m 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5863475285
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5863110116)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (7effeab5cacbce8a08f506170f2e2a9ec58e39df) to ed43e69 (merge-base) [diff](https://github.com/apache/datafusion/compare/ed43e69fdca8737c9d065b53b5fc5e4183302775..7effeab5cacbce8a08f506170f2e2a9ec58e39df)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.28 108.0Β±0.72ms ? ?/sec 1.00 84.3Β±0.37ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.08 102.2Β±0.40ms ? ?/sec 1.00 94.4Β±0.45ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.07 100.4Β±0.73ms ? ?/sec 1.00 94.2Β±0.54ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.00 127.7Β±1.67ms ? ?/sec 1.00 128.0Β±0.85ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.01 134.6Β±3.72ms ? ?/sec 1.00 132.9Β±3.93ms ? ?/sec
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only for benchmarks that write a results JSON (the `dfbench` suites, and the suites `bench.sh` runs through the Criterion SQL harness since apache/datafusion#25644), and only under a memory limit: `DATAFUSION_RUNTIME_MEMORY_LIMIT`, or one the suite sets itself.
Base: `ed43e69 (merge-base)` | Changed: `spill-dedup-view-arrays`
**`spill_views`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| spill_views/q01_sort_string_1_distinct_repeated | 40.6 MiB | 40.6 MiB | +0.0% |
| spill_views/q02_sort_string_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q03_sort_binary_1000_distinct_repeated | 39.8 MiB | 39.8 MiB | +0.0% |
| spill_views/q04_sort_string_all_distinct_distinct | 102.6 MiB | 102.6 MiB | +0.0% |
| spill_views/q05_group_by_string_all_distinct_distinct | 96.0 MiB | 96.0 MiB | +0.0% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| spill_views | base (`ed43e69 (merge-base)`) | 102.6 MiB | 755.5 MiB | 652.9 MiB | 7.4Γ |
| spill_views | changed (`spill-dedup-view-arrays`) | 102.6 MiB | 747.9 MiB | 645.3 MiB | 7.3Γ |
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 670.3s |
| Peak memory | 755.5 MiB |
| Avg memory | 33.0 MiB |
| CPU user | 63.3s |
| CPU sys | 29.7s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 660.2s |
| Peak memory | 747.9 MiB |
| Avg memory | 39.9 MiB |
| CPU user | 78.9s |
| CPU sys | 21.9s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783844490
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783727326)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783727326-2604-wfkl2 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783823977
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783725982-2601-z4b9q 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784340158
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784307832)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5784307832-2617-gx8mq 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789125588
run benchmark spill_views
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789428527
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789127530)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.88 ms β 134.69 ms β no change β
β Q2 β 136.58 ms β 136.08 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 199.80 ms β 197.14 ms β no change β
β Q5 β 230.33 ms β 230.58 ms β no change β
β Q6 β 253.14 ms β 256.31 ms β no change β
β Q7 β 686.03 ms β 744.64 ms β 1.09x slower β
β Q8 β 490.50 ms β 515.45 ms β 1.05x slower β
β Q9 β 524.56 ms β 526.46 ms β no change β
β Q10 β 1237.58 ms β 1241.89 ms β no change β
β Q11 β 375.02 ms β 376.79 ms β no change β
βββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4268.42ms β
β Total Time (spill-dedup-view-arrays) β 4360.03ms β
β Average Time (HEAD) β 426.84ms β
β Average Time (spill-dedup-view-arrays) β 436.00ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 8 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark sort_tpch1.json
--------------------
βββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Q1 β 134.88 / 135.69 Β±1.20 / 138.07 ms β 134.69 / 135.63 Β±1.15 / 137.84 ms β no change β
β Q2 β 136.58 / 138.50 Β±1.18 / 139.80 ms β 136.08 / 138.28 Β±1.90 / 141.03 ms β no change β
β Q3 β FAIL β FAIL β incomparable β
β Q4 β 199.80 / 205.65 Β±3.98 / 209.46 ms β 197.14 / 203.76 Β±5.47 / 211.05 ms β no change β
β Q5 β 230.33 / 235.25 Β±2.95 / 238.51 ms β 230.58 / 233.03 Β±1.34 / 234.42 ms β no change β
β Q6 β 253.14 / 254.08 Β±0.72 / 254.84 ms β 256.31 / 258.79 Β±1.62 / 261.00 ms β no change β
β Q7 β 686.03 / 695.39 Β±5.43 / 702.69 ms β 744.64 / 746.74 Β±2.23 / 750.70 ms β 1.07x slower β
β Q8 β 490.50 / 495.11 Β±4.94 / 504.22 ms β 515.45 / 522.30 Β±4.44 / 528.25 ms β 1.05x slower β
β Q9 β 524.56 / 537.16 Β±7.76 / 545.35 ms β 526.46 / 551.55 Β±19.80 / 579.70 ms β no change β
β Q10 β 1237.58 / 1278.57 Β±32.80 / 1319.35 ms β 1241.89 / 1271.49 Β±29.03 / 1322.15 ms β no change β
β Q11 β 375.02 / 381.27 Β±4.37 / 388.40 ms β 376.79 / 385.62 Β±5.29 / 392.66 ms β no change β
βββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Benchmark Summary β β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 4356.67ms β
β Total Time (spill-dedup-view-arrays) β 4447.20ms β
β Average Time (HEAD) β 435.67ms β
β Average Time (spill-dedup-view-arrays) β 444.72ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 8 β
β Queries with Failure β 1 β
ββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `207d7f3 (merge-base)` | Changed: `207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67`
**`sort_tpch` β `sort_tpch1`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| 1 | 175.1 MiB | 174.9 MiB | -0.1% |
| 2 | 219.4 MiB | 217.7 MiB | -0.8% |
| 3 | 518.4 MiB | 516.0 MiB | -0.5% |
| 4 | 264.6 MiB | 266.1 MiB | +0.6% |
| 5 | 294.0 MiB | 294.0 MiB | +0.0% |
| 6 | 355.3 MiB | 356.5 MiB | +0.3% |
| 7 | 509.0 MiB | 509.0 MiB | +0.0% |
| 8 | 508.7 MiB | 508.7 MiB | +0.0% |
| 9 | 497.9 MiB | 501.2 MiB | +0.7% |
| 10 | 509.1 MiB | 506.0 MiB | -0.6% |
| 11 | 523.8 MiB | 522.6 MiB | -0.2% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| sort_tpch | base (`207d7f3 (merge-base)`) | 523.8 MiB | 1.1 GiB | 648.2 MiB | 2.2Γ |
| sort_tpch | changed (`207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67`) | 522.6 MiB | 1.2 GiB | 748.1 MiB | 2.4Γ |
</details>
<details><summary>Resource Usage</summary>
**sort_tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.1 GiB |
| Avg memory | 822.6 MiB |
| CPU user | 65.3s |
| CPU sys | 16.7s |
| Peak spill | 975.2 MiB |
**sort_tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 25.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 869.5 MiB |
| CPU user | 65.2s |
| CPU sys | 16.9s |
| Peak spill | 1.0 GiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789379980
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789124786)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67 (207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
SPILL_VIEWS_LIMIT_DISTINCT: "128M"
changed:
ref: "207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q01_sort_string_1_distinct_repeated 1.00 109.0Β±1.52ms ? ?/sec 1.01 109.8Β±1.02ms ? ?/sec
spill_views/q02_sort_string_1000_distinct_repeated 1.00 103.9Β±1.37ms ? ?/sec 1.01 105.3Β±0.72ms ? ?/sec
spill_views/q03_sort_binary_1000_distinct_repeated 1.00 103.5Β±1.00ms ? ?/sec 1.00 103.3Β±1.10ms ? ?/sec
spill_views/q04_sort_string_all_distinct_distinct 1.01 135.1Β±2.40ms ? ?/sec 1.00 133.5Β±2.95ms ? ?/sec
spill_views/q05_group_by_string_all_distinct_distinct 1.00 116.1Β±1.62ms ? ?/sec 1.00 115.9Β±1.30ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 575.3s |
| Peak memory | 824.5 MiB |
| Avg memory | 39.8 MiB |
| CPU user | 70.1s |
| CPU sys | 30.1s |
| Peak spill | 84.2 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 585.2s |
| Peak memory | 807.3 MiB |
| Avg memory | 39.0 MiB |
| CPU user | 68.2s |
| CPU sys | 30.4s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789385427
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5789126649)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5789126649-2628-xdd24 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (2918b059cbade7777d28a7b7b1c65b82c130f5b6) to 207d7f3 (merge-base) [diff](https://github.com/apache/datafusion/compare/207d7f3c2721fb48ba4d73dd8d22b1b324c6fe67..2918b059cbade7777d28a7b7b1c65b82c130f5b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark sort_tpch
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5789768398
## Benchmark bot results after rebase (head `2918b059cb`, base `207d7f3c27`)
A/B = main vs this PR, A/A = main vs main. Ratios are branch / base (>1.00 = slower with this PR).
### spill_views with `SPILL_VIEWS_LIMIT_DISTINCT=128M` (q05 now finishes on both sides)
| Query | A/B run 1 | A/B run 2 | A/A run 1 | A/A run 2 |
| --- | --- | --- | --- | --- |
| q01 sort, 1 distinct value | 105.5 β 88.4 ms (**0.84x**) | 105.3 β 90.4 ms (**0.86x**) | 1.09x | 1.01x |
| q02 sort, 1000 distinct values | 0.98x | 0.99x | 1.00x | 1.01x |
| q03 sort, BinaryView, 1000 distinct | 1.00x | 0.97x | 1.01x | 1.00x |
| q04 sort, all distinct | 126.6 β 180.7 ms (**1.43x**) | 126.3 β 174.3 ms (**1.38x**) | 1.03x | 0.99x |
| q05 GROUP BY, all distinct | 1.05x | 0.99x | 0.96x | 1.00x |
### spill_views at the default limits
| | A/B runs | Result |
| --- | --- | --- |
| This round | 1 | q05 fails on the branch side at 96M |
| All rounds | 5 | 5 failed, always on the branch side; A/A 0 of 3 failed |
### sort_tpch, 512M limit, 4 partitions
| Query | A/B | A/A |
| --- | --- | --- |
| Q8 (`l_comment`) | 497 β 514 ms (1.03x) | 495 β 522 ms (1.05x) |
| Q9 (`l_comment`) | 539 β 568 ms (1.05x) | 537 β 552 ms (1.03x) |
| Q11 (`l_comment`) | 381 β 418 ms (**1.10x**) | 381 β 386 ms (1.01x) |
| Total | 4316 β 4423 ms (1.02x) | 4357 β 4447 ms (1.02x) |
The A/A run was noisier this round, so only Q11 stands out. It matches the earlier runs (1.11x and 1.08x).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783081490
run benchmark sort_tpch spill_views
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783079245
run benchmark sort_tpch spill_views
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783287906
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783075232)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.74 ms β 41.48 ms β no change β
β QQuery 2 β 19.27 ms β 19.01 ms β no change β
β QQuery 3 β 29.65 ms β 28.96 ms β no change β
β QQuery 4 β 18.06 ms β 17.79 ms β no change β
β QQuery 5 β 36.58 ms β 36.45 ms β no change β
β QQuery 6 β 17.00 ms β 16.98 ms β no change β
β QQuery 7 β 43.02 ms β 43.81 ms β no change β
β QQuery 8 β 40.84 ms β 41.16 ms β no change β
β QQuery 9 β 49.90 ms β 48.80 ms β no change β
β QQuery 10 β 41.58 ms β 40.87 ms β no change β
β QQuery 11 β 14.54 ms β 13.53 ms β +1.07x faster β
β QQuery 12 β 21.85 ms β 21.23 ms β no change β
β QQuery 13 β 42.89 ms β 40.26 ms β +1.07x faster β
β QQuery 14 β 25.31 ms β 24.51 ms β no change β
β QQuery 15 β 31.99 ms β 31.39 ms β no change β
β QQuery 16 β 14.46 ms β 14.35 ms β no change β
β QQuery 17 β 78.18 ms β 78.59 ms β no change β
β QQuery 18 β 62.97 ms β 63.73 ms β no change β
β QQuery 19 β 32.61 ms β 32.97 ms β no change β
β QQuery 20 β 33.42 ms β 33.20 ms β no change β
β QQuery 21 β 57.13 ms β 58.81 ms β no change β
β QQuery 22 β 14.95 ms β 14.80 ms β no change β
βββββββββββββ΄βββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 767.94ms β
β Total Time (spill-dedup-view-arrays) β 762.70ms β
β Average Time (HEAD) β 34.91ms β
β Average Time (spill-dedup-view-arrays) β 34.67ms β
β Queries Faster β 2 β
β Queries Slower β 0 β
β Queries with No Change β 20 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark tpch_sf1.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 1 β 41.74 / 43.12 Β±1.59 / 45.35 ms β 41.48 / 42.89 Β±1.22 / 44.70 ms β no change β
β QQuery 2 β 19.27 / 19.81 Β±0.41 / 20.54 ms β 19.01 / 19.24 Β±0.28 / 19.78 ms β no change β
β QQuery 3 β 29.65 / 29.84 Β±0.19 / 30.20 ms β 28.96 / 29.36 Β±0.25 / 29.69 ms β no change β
β QQuery 4 β 18.06 / 18.37 Β±0.43 / 19.22 ms β 17.79 / 18.17 Β±0.47 / 19.09 ms β no change β
β QQuery 5 β 36.58 / 38.00 Β±1.50 / 40.84 ms β 36.45 / 36.78 Β±0.20 / 36.99 ms β no change β
β QQuery 6 β 17.00 / 17.35 Β±0.28 / 17.85 ms β 16.98 / 17.23 Β±0.21 / 17.55 ms β no change β
β QQuery 7 β 43.02 / 44.35 Β±1.41 / 46.67 ms β 43.81 / 46.03 Β±2.29 / 49.93 ms β no change β
β QQuery 8 β 40.84 / 41.89 Β±0.82 / 43.25 ms β 41.16 / 41.36 Β±0.19 / 41.59 ms β no change β
β QQuery 9 β 49.90 / 51.22 Β±1.25 / 53.52 ms β 48.80 / 50.25 Β±1.49 / 52.90 ms β no change β
β QQuery 10 β 41.58 / 42.18 Β±0.53 / 42.91 ms β 40.87 / 41.82 Β±1.04 / 43.78 ms β no change β
β QQuery 11 β 14.54 / 14.60 Β±0.07 / 14.73 ms β 13.53 / 13.85 Β±0.30 / 14.34 ms β +1.05x faster β
β QQuery 12 β 21.85 / 22.52 Β±0.61 / 23.61 ms β 21.23 / 21.57 Β±0.30 / 22.14 ms β no change β
β QQuery 13 β 42.89 / 44.60 Β±2.02 / 48.38 ms β 40.26 / 41.24 Β±0.94 / 43.00 ms β +1.08x faster β
β QQuery 14 β 25.31 / 25.97 Β±0.95 / 27.85 ms β 24.51 / 24.64 Β±0.10 / 24.78 ms β +1.05x faster β
β QQuery 15 β 31.99 / 32.50 Β±0.51 / 33.46 ms β 31.39 / 31.90 Β±0.82 / 33.54 ms β no change β
β QQuery 16 β 14.46 / 14.74 Β±0.16 / 14.95 ms β 14.35 / 14.48 Β±0.11 / 14.67 ms β no change β
β QQuery 17 β 78.18 / 79.81 Β±1.38 / 81.74 ms β 78.59 / 79.51 Β±0.68 / 80.70 ms β no change β
β QQuery 18 β 62.97 / 66.68 Β±3.97 / 72.87 ms β 63.73 / 66.40 Β±2.72 / 70.51 ms β no change β
β QQuery 19 β 32.61 / 34.07 Β±1.21 / 35.82 ms β 32.97 / 33.13 Β±0.16 / 33.38 ms β no change β
β QQuery 20 β 33.42 / 34.21 Β±1.13 / 36.44 ms β 33.20 / 33.46 Β±0.18 / 33.76 ms β no change β
β QQuery 21 β 57.13 / 58.32 Β±1.13 / 60.45 ms β 58.81 / 59.38 Β±0.67 / 60.60 ms β no change β
β QQuery 22 β 14.95 / 16.43 Β±2.07 / 20.47 ms β 14.80 / 14.95 Β±0.14 / 15.15 ms β +1.10x faster β
βββββββββββββ΄βββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 790.56ms β
β Total Time (spill-dedup-view-arrays) β 777.65ms β
β Average Time (HEAD) β 35.93ms β
β Average Time (spill-dedup-view-arrays) β 35.35ms β
β Queries Faster β 4 β
β Queries Slower β 0 β
β Queries with No Change β 18 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 525.0 MiB |
| CPU user | 22.2s |
| CPU sys | 2.0s |
| Peak spill | 0 B |
**tpch β branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 516.1 MiB |
| CPU user | 22.1s |
| CPU sys | 1.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783350095
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783081490)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783081490-2593-gwklk 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
DATAFUSION_EXECUTION_TARGET_PARTITIONS: "4"
DATAFUSION_RUNTIME_MEMORY_LIMIT: "512M"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783758546
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783724486)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5783724486-2600-2lzlf 6.12.94+ #1 SMP Wed Aug 19 07:47:20 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing spill-dedup-view-arrays (cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..cbdecca5ea24c0e9fb2a5bd56d156d6ec18b66ee)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783954673
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5783725982)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.25 ms β 1.26 ms β no change β
β QQuery 1 β 11.87 ms β 11.58 ms β no change β
β QQuery 2 β 37.17 ms β 36.89 ms β no change β
β QQuery 3 β 32.68 ms β 31.34 ms β no change β
β QQuery 4 β 257.08 ms β 238.99 ms β +1.08x faster β
β QQuery 5 β 290.10 ms β 289.94 ms β no change β
β QQuery 6 β 1.47 ms β 1.33 ms β +1.10x faster β
β QQuery 7 β 13.89 ms β 13.58 ms β no change β
β QQuery 8 β 338.21 ms β 357.62 ms β 1.06x slower β
β QQuery 9 β 496.63 ms β 509.27 ms β no change β
β QQuery 10 β 67.26 ms β 64.14 ms β no change β
β QQuery 11 β 78.83 ms β 78.27 ms β no change β
β QQuery 12 β 272.41 ms β 264.51 ms β no change β
β QQuery 13 β 372.94 ms β 383.32 ms β no change β
β QQuery 14 β 280.90 ms β 297.55 ms β 1.06x slower β
β QQuery 15 β 289.25 ms β 298.57 ms β no change β
β QQuery 16 β 624.62 ms β 640.69 ms β no change β
β QQuery 17 β 630.46 ms β 646.67 ms β no change β
β QQuery 18 β 1340.61 ms β 1296.40 ms β no change β
β QQuery 19 β 27.79 ms β 28.09 ms β no change β
β QQuery 20 β 519.29 ms β 524.53 ms β no change β
β QQuery 21 β 522.52 ms β 524.35 ms β no change β
β QQuery 22 β 1000.95 ms β 1039.25 ms β no change β
β QQuery 23 β 3096.52 ms β 3307.54 ms β 1.07x slower β
β QQuery 24 β 42.63 ms β 41.42 ms β no change β
β QQuery 25 β 105.15 ms β 106.29 ms β no change β
β QQuery 26 β 41.07 ms β 41.44 ms β no change β
β QQuery 27 β 522.60 ms β 515.45 ms β no change β
β QQuery 28 β 2958.28 ms β 2944.19 ms β no change β
β QQuery 29 β 41.40 ms β 42.79 ms β no change β
β QQuery 30 β 303.75 ms β 320.69 ms β 1.06x slower β
β QQuery 31 β 290.12 ms β 307.34 ms β 1.06x slower β
β QQuery 32 β 1183.96 ms β 1159.10 ms β no change β
β QQuery 33 β 1647.88 ms β 1633.32 ms β no change β
β QQuery 34 β 1639.02 ms β 1682.45 ms β no change β
β QQuery 35 β 296.52 ms β 294.31 ms β no change β
β QQuery 36 β 67.33 ms β 65.02 ms β no change β
β QQuery 37 β 35.00 ms β 34.80 ms β no change β
β QQuery 38 β 42.44 ms β 41.25 ms β no change β
β QQuery 39 β 134.93 ms β 144.96 ms β 1.07x slower β
β QQuery 40 β 14.27 ms β 15.45 ms β 1.08x slower β
β QQuery 41 β 13.85 ms β 14.70 ms β 1.06x slower β
β QQuery 42 β 14.66 ms β 14.32 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 19999.56ms β
β Total Time (spill-dedup-view-arrays) β 20304.95ms β
β Average Time (HEAD) β 465.11ms β
β Average Time (spill-dedup-view-arrays) β 472.21ms β
β Queries Faster β 2 β
β Queries Slower β 8 β
β Queries with No Change β 33 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and spill-dedup-view-arrays
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β spill-dedup-view-arrays β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.25 / 4.04 Β±5.48 / 15.00 ms β 1.26 / 4.04 Β±5.48 / 15.01 ms β no change β
β QQuery 1 β 11.87 / 12.09 Β±0.19 / 12.44 ms β 11.58 / 11.80 Β±0.12 / 11.95 ms β no change β
β QQuery 2 β 37.17 / 37.39 Β±0.18 / 37.62 ms β 36.89 / 37.30 Β±0.43 / 38.11 ms β no change β
β QQuery 3 β 32.68 / 33.77 Β±1.16 / 36.00 ms β 31.34 / 31.57 Β±0.12 / 31.69 ms β +1.07x faster β
β QQuery 4 β 257.08 / 271.18 Β±8.86 / 283.19 ms β 238.99 / 259.70 Β±13.42 / 272.88 ms β no change β
β QQuery 5 β 290.10 / 298.76 Β±8.69 / 314.37 ms β 289.94 / 294.86 Β±2.64 / 297.34 ms β no change β
β QQuery 6 β 1.47 / 1.75 Β±0.29 / 2.16 ms β 1.33 / 1.48 Β±0.23 / 1.94 ms β +1.18x faster β
β QQuery 7 β 13.89 / 14.13 Β±0.13 / 14.24 ms β 13.58 / 13.88 Β±0.24 / 14.15 ms β no change β
β QQuery 8 β 338.21 / 364.17 Β±20.88 / 396.43 ms β 357.62 / 379.34 Β±20.72 / 411.13 ms β no change β
β QQuery 9 β 496.63 / 521.11 Β±14.53 / 539.52 ms β 509.27 / 529.11 Β±22.94 / 570.51 ms β no change β
β QQuery 10 β 67.26 / 68.05 Β±0.51 / 68.85 ms β 64.14 / 68.85 Β±7.97 / 84.72 ms β no change β
β QQuery 11 β 78.83 / 83.41 Β±5.09 / 93.15 ms β 78.27 / 81.45 Β±1.63 / 82.78 ms β no change β
β QQuery 12 β 272.41 / 305.71 Β±18.07 / 326.60 ms β 264.51 / 276.86 Β±12.32 / 297.47 ms β +1.10x faster β
β QQuery 13 β 372.94 / 389.14 Β±11.60 / 404.73 ms β 383.32 / 403.07 Β±21.31 / 435.96 ms β no change β
β QQuery 14 β 280.90 / 295.09 Β±15.52 / 319.27 ms β 297.55 / 306.48 Β±8.75 / 320.04 ms β no change β
β QQuery 15 β 289.25 / 309.53 Β±22.58 / 340.69 ms β 298.57 / 312.09 Β±14.20 / 338.16 ms β no change β
β QQuery 16 β 624.62 / 646.41 Β±17.89 / 673.23 ms β 640.69 / 663.44 Β±14.09 / 678.57 ms β no change β
β QQuery 17 β 630.46 / 659.76 Β±17.38 / 684.44 ms β 646.67 / 682.70 Β±31.17 / 728.96 ms β no change β
β QQuery 18 β 1340.61 / 1387.50 Β±51.20 / 1482.71 ms β 1296.40 / 1329.16 Β±31.93 / 1388.85 ms β no change β
β QQuery 19 β 27.79 / 31.57 Β±3.65 / 38.33 ms β 28.09 / 45.39 Β±15.00 / 68.71 ms β 1.44x slower β
β QQuery 20 β 519.29 / 533.79 Β±12.12 / 553.31 ms β 524.53 / 532.28 Β±6.56 / 540.40 ms β no change β
β QQuery 21 β 522.52 / 533.01 Β±8.35 / 542.89 ms β 524.35 / 538.66 Β±12.77 / 560.48 ms β no change β
β QQuery 22 β 1000.95 / 1039.71 Β±31.22 / 1092.10 ms β 1039.25 / 1066.12 Β±23.89 / 1109.38 ms β no change β
β QQuery 23 β 3096.52 / 3360.02 Β±164.79 / 3590.50 ms β 3307.54 / 3407.14 Β±90.14 / 3551.73 ms β no change β
β QQuery 24 β 42.63 / 132.10 Β±114.70 / 353.92 ms β 41.42 / 200.85 Β±272.91 / 742.00 ms β 1.52x slower β
β QQuery 25 β 105.15 / 109.88 Β±3.82 / 115.52 ms β 106.29 / 109.72 Β±2.12 / 112.91 ms β no change β
β QQuery 26 β 41.07 / 42.71 Β±2.00 / 46.49 ms β 41.44 / 42.46 Β±1.35 / 45.10 ms β no change β
β QQuery 27 β 522.60 / 529.06 Β±6.39 / 541.25 ms β 515.45 / 530.60 Β±11.93 / 544.09 ms β no change β
β QQuery 28 β 2958.28 / 2994.73 Β±44.66 / 3080.88 ms β 2944.19 / 3002.47 Β±50.34 / 3091.66 ms β no change β
β QQuery 29 β 41.40 / 55.69 Β±22.64 / 100.42 ms β 42.79 / 47.33 Β±8.64 / 64.60 ms β +1.18x faster β
β QQuery 30 β 303.75 / 327.53 Β±19.46 / 357.25 ms β 320.69 / 332.72 Β±11.67 / 352.70 ms β no change β
β QQuery 31 β 290.12 / 304.17 Β±15.63 / 332.63 ms β 307.34 / 323.33 Β±21.06 / 363.21 ms β 1.06x slower β
β QQuery 32 β 1183.96 / 1284.93 Β±92.81 / 1438.47 ms β 1159.10 / 1284.94 Β±71.58 / 1369.14 ms β no change β
β QQuery 33 β 1647.88 / 1717.06 Β±83.86 / 1881.06 ms β 1633.32 / 1743.93 Β±86.48 / 1893.22 ms β no change β
β QQuery 34 β 1639.02 / 1723.90 Β±85.97 / 1868.84 ms β 1682.45 / 1842.16 Β±121.42 / 1977.24 ms β 1.07x slower β
β QQuery 35 β 296.52 / 365.51 Β±67.72 / 484.63 ms β 294.31 / 419.86 Β±196.60 / 811.40 ms β 1.15x slower β
β QQuery 36 β 67.33 / 79.77 Β±12.41 / 103.11 ms β 65.02 / 68.09 Β±2.10 / 70.75 ms β +1.17x faster β
β QQuery 37 β 35.00 / 36.10 Β±0.65 / 36.93 ms β 34.80 / 35.33 Β±0.29 / 35.57 ms β no change β
β QQuery 38 β 42.44 / 51.38 Β±5.90 / 58.99 ms β 41.25 / 49.27 Β±9.17 / 65.87 ms β no change β
β QQuery 39 β 134.93 / 153.91 Β±14.67 / 174.35 ms β 144.96 / 154.69 Β±8.69 / 166.15 ms β no change β
β QQuery 40 β 14.27 / 17.74 Β±5.51 / 28.71 ms β 15.45 / 15.99 Β±0.57 / 17.08 ms β +1.11x faster β
β QQuery 41 β 13.85 / 14.74 Β±0.67 / 15.41 ms β 14.70 / 14.99 Β±0.20 / 15.32 ms β no change β
β QQuery 42 β 14.66 / 14.93 Β±0.24 / 15.30 ms β 14.32 / 23.15 Β±14.22 / 51.16 ms β 1.55x slower β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 21156.94ms β
β Total Time (spill-dedup-view-arrays) β 21518.67ms β
β Average Time (HEAD) β 492.02ms β
β Average Time (spill-dedup-view-arrays) β 500.43ms β
β Queries Faster β 6 β
β Queries Slower β 6 β
β Queries with No Change β 31 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 18.4 GiB |
| Avg memory | 6.1 GiB |
| CPU user | 1037.3s |
| CPU sys | 106.6s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 110.0s |
| Peak memory | 17.9 GiB |
| Avg memory | 6.0 GiB |
| CPU user | 1036.5s |
| CPU sys | 112.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5783906160
run benchmark spill_views
env:
QUERY: "5"
changed:
ref: 5b5bed97f5fdbbb9a7026653fab4b21170a7a856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784131606
Benchmark for [this request](https://github.com/apache/datafusion/pull/23565#issuecomment-5783733220) failed before finishing (Kubernetes reason: `BackoffLimitExceeded`).
Benchmarks requested: `spill_views`
<details><summary>Runner log (last 40 lines)</summary>
```
Compiling datafusion-datasource-arrow v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-arrow)
Compiling datafusion-datasource-json v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-json)
Compiling datafusion-datasource-parquet v55.1.0 (/workspace/datafusion-branch/datafusion/datasource-parquet)
Compiling datafusion-physical-optimizer v55.1.0 (/workspace/datafusion-branch/datafusion/physical-optimizer)
Compiling datafusion-catalog-listing v55.1.0 (/workspace/datafusion-branch/datafusion/catalog-listing)
Compiling datafusion-functions-table v55.1.0 (/workspace/datafusion-branch/datafusion/functions-table)
Compiling datafusion v55.1.0 (/workspace/datafusion-branch/datafusion/core)
Compiling datafusion-proto v55.1.0 (/workspace/datafusion-branch/datafusion/proto)
Compiling datafusion-benchmarks v55.1.0 (/workspace/datafusion-branch/benchmarks)
Finished `bench` profile [optimized] target(s) in 5m 13s
Running benches/sql.rs (/workspace/datafusion-branch/target/release/deps/sql-2af2193982da74c5)
Gnuplot not found, using plotters backend
Benchmarking spill_views/q01_sort_string_1_distinct_repeated
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Collecting 10 samples in estimated 5.2711 s (60 iterations)
Benchmarking spill_views/q01_sort_string_1_distinct_repeated: Analyzing
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Collecting 10 samples in estimated 5.8152 s (60 iterations)
Benchmarking spill_views/q02_sort_string_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Warming up for 3.0000 s
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Collecting 10 samples in estimated 5.8387 s (60 iterations)
Benchmarking spill_views/q03_sort_binary_1000_distinct_repeated: Analyzing
Benchmarking spill_views/q04_sort_string_all_distinct_distinct
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Collecting 10 samples in estimated 6.5801 s (40 iterations)
Benchmarking spill_views/q04_sort_string_all_distinct_distinct: Analyzing
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Warming up for 3.0000 s
Benchmarking spill_views/q05_group_by_string_all_distinct_distinct: Collecting 10 samples in estimated 5.8409 s (40 iterations)
thread 'main' (15081) panicked at benchmarks/src/sql_benchmark_runner.rs:224:25:
Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted("Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\n FinalHashAggregateStream[1]#6496(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[2]#6497(can spill: true) consumed 29.3 MB, peak 29.3 MB,\n FinalHashAggregateStream[0]#6506(can spill: true) consumed 27.1 MB, peak 27.1 MB,\n RepartitionExec[0]#6502(can spill: true) consumed 4.8 MB, peak 4.8 MB,\n RepartitionExec[2]#6504(can spill: true) consumed 2.4 MB, peak 2.4 MB.\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)")
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'main' (15081) panicked at benchmarks/benches/sql.rs:55:27:
failed to run SQL benchmarks: Execution("criterion benchmark failed: Failed to run benchmark q05_group_by_string_all_distinct_distinct: ResourcesExhausted(\"Additional allocation failed for FinalHashAggregateStream[3] with top memory consumers (across reservations) as:\\n FinalHashAggregateStream[1]#6496(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[2]#6497(can spill: true) consumed 29.3 MB, peak 29.3 MB,\\n FinalHashAggregateStream[0]#6506(can spill: true) consumed 27.1 MB, peak 27.1 MB,\\n RepartitionExec[0]#6502(can spill: true) consumed 4.8 MB, peak 4.8 MB,\\n RepartitionExec[2]#6504(can spill: true) consumed 2.4 MB, peak 2.4 MB.\\nError: Failed to allocate additional 2.3 MB for FinalHashAggregateStream[3] with 0.0 B already allocated for this reservation - 2.1 MB remain available for the total memory pool: greedy(used: 93.9 MB, pool_size: 96.0 MB)\")")
error: bench failed, to rerun pass `--bench sql`
```
</details>
<details><summary>Kubernetes message</summary>
```
Job has reached the specified backoff limit
```
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] GC & Deduplicate String View on Spill [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#issuecomment-5784539544
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/23565#issuecomment-5784307832)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing 5b5bed97f5fdbbb9a7026653fab4b21170a7a856 (5b5bed97f5fdbbb9a7026653fab4b21170a7a856) to 5b5bed9 (merge-base) [diff](https://github.com/apache/datafusion/compare/5b5bed97f5fdbbb9a7026653fab4b21170a7a856..5b5bed97f5fdbbb9a7026653fab4b21170a7a856)
<details><summary>Run configuration</summary>
```yaml
run benchmark spill_views
env:
BENCH_QUERY: "4"
changed:
ref: "5b5bed97f5fdbbb9a7026653fab4b21170a7a856"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
group HEAD spill-dedup-view-arrays
----- ---- -----------------------
spill_views/q04_sort_string_all_distinct_distinct 1.06 128.0Β±5.49ms ? ?/sec 1.00 120.4Β±2.13ms ? ?/sec
```
</p>
</details>
<details><summary>Resource Usage</summary>
**spill_views β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 335.2s |
| Peak memory | 404.1 MiB |
| Avg memory | 12.9 MiB |
| CPU user | 8.7s |
| CPU sys | 5.0s |
| Peak spill | 52.4 MiB |
**spill_views β branch**
| Metric | Value |
|--------|-------|
| Wall time | 340.2s |
| Peak memory | 414.1 MiB |
| Avg memory | 14.2 MiB |
| CPU user | 10.0s |
| CPU sys | 5.8s |
| Peak spill | 84.2 MiB |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org