You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/01 19:07:19 UTC
[PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
adriangb opened a new pull request, #24859:
URL: https://github.com/apache/datafusion/pull/24859
## Which issue does this PR close?
No existing issue. This was found while investigating a production out of memory. Happy to file one if you would like it tracked for the changelog.
## Rationale for this change
`SingleDistinctToGroupBy` rewrites `AGG(DISTINCT x)` into a two phase group by, which keeps a high cardinality distinct off the one-boxed-accumulator-per-group path in `GroupsAccumulatorAdapter`. The rule already tolerates a non-distinct `sum`, `min` or `max` next to the distinct aggregate, but bails out on `count`, so the very common
```sql
SELECT g, count(*), count(DISTINCT x) FROM t GROUP BY g
```
shape keeps the unrewritten plan and its memory profile.
We hit this in production: a query of exactly that shape drove a process running DataFusion to 10.98 GB and death, and the single `count(*)` was the only reason the rewrite did not apply.
## What changes are included in this PR?
This allows a non-distinct `count` as well. `count` is the one supported function whose outer phase is a *different* function: the inner group by counts the rows of each `(group, distinct value)` partition, and the outer phase adds those partial counts up with `sum`, since `count` over a group is the sum of the counts of any partition of that group.
Two details follow from the substitution:
- `count` and `sum` come from the session function registry (as `replace_distinct_aggregate` already does for `first_value`), and the rewrite only fires when the aggregate is that exact `count`, compared by identity rather than by name. A session without a registry, or with its own `count`, is left alone.
- `count` returns a non-null `0` over an empty input while `sum` of no rows is NULL, which is reachable for an aggregate with no `GROUP BY`: the inner aggregate emits no rows and the outer still emits one, so `SELECT count(*), count(DISTINCT x) FROM empty` would return `NULL, 0` instead of `0, 0`. The projection selects `CASE WHEN sum(alias) IS NOT NULL THEN sum(alias) ELSE 0 END`, restoring the `0` and keeping the column's type and nullability as `count` had them.
`FILTER` and `ORDER BY` still block the rewrite.
Files touched beyond the rule itself:
- `datafusion/sqllogictest/test_files/single_distinct_to_groupby.slt`: the new coverage described below.
- `datafusion/sqllogictest/test_files/clickbench.slt`: the one existing snapshot in the repository that changes, discussed below.
- `datafusion/substrait/tests/cases/roundtrip_logical_plan.rs`: `aggregate_distinct_with_having` now builds its session with this rule removed, so it keeps round tripping the plan shape the test was written for.
## What is the testing strategy for this PR?
`single_distinct_to_groupby.slt` asserts every result twice, once under `datafusion.optimizer.max_passes = 0` and once under the default, with identical expected blocks, so a null-handling or type error surfaces as a result mismatch rather than only a plan diff. It covers `count(*)` vs `count(1)` vs `count(col)` grouped and ungrouped, a group whose distinct column is entirely NULL, a group with NULLs in both the distinct and summed columns, empty input three ways, `HAVING` plus `ORDER BY` on the rewritten count, and the production join shape.
Exactly one existing snapshot in the repository changes: the ClickBench Q22 `EXPLAIN`, which is this shape verbatim. Its result block directly beneath, running on real ClickBench parquet, is unchanged. The physical `SortExec: TopK(fetch=10)` moves from below the projection to above it, because the sort key is now a `CASE` output rather than a raw aggregate column. That is order-equivalent, since the `CASE` is the identity on every non-NULL input and the `sum` is never NULL in a grouped aggregate.
Run locally: the full sqllogictest suite, plus `datafusion --test core_integration` (1079), `--test tpcds_planning` (198) and `-p datafusion --lib` (444). `cargo clippy -p datafusion-optimizer --all-targets` is clean.
### Benchmarks
Q22 is the only ClickBench query whose plan changes. Q9 (`RegionID, SUM, COUNT(*), AVG, COUNT(DISTINCT UserID)`) still bails out, because `AVG` disqualifies it.
Measured on `clickbench_partitioned` (100 files, ~100M rows), release builds of this branch and of the base commit it sat on at the time of the run, on a 12-core machine.
A run-level A/B could not resolve a change this small here. Comparing the base binary against *itself* with `compare.py` reported 9 queries faster, 28 slower and 6 unchanged, with swings up to 1.58x, and two runs of the same base-vs-branch comparison gave opposite verdicts (11 faster / 21 slower, then 30 faster / 6 slower, with a 1.97x swing). Those tables measure background load, not the patch, because one arm is a full 43-query pass of about four minutes and load drifts between the arms.
Instead the arms were paired per query, running base and branch back to back and alternating which goes first, over 40 repetitions. The 42 queries whose plans are unchanged then serve as an in-experiment control for residual bias.
Q22, net of the control bias (difference in differences, bootstrap CI):
```
-2.03% 95% CI [-5.48%, +1.39%]
```
The interval includes zero, so there is no measurable latency difference, and the 95% upper bound excludes a Q22 regression larger than about 1.5%. Pooled controls moved +0.36% [-0.64%, +1.05%], confirming the setup resolves effects of roughly 3% and no better.
This is latency-neutral on ClickBench, consistent with #11360, which found removing the rule entirely to be a wash. The case for the change rests on the memory behaviour of the rewritten plan, not on latency.
**Not covered: memory.** These runs used no `--memory-limit`, so they do not exercise the spilling behaviour that motivates the rewrite. That is a separate experiment.
## Are there any user-facing changes?
No public API change and no change to query results. Plans for `SELECT ..., count(...), count(DISTINCT x) ... GROUP BY ...` change shape, so `EXPLAIN` output for that shape differs, and such queries should use substantially less memory. The ClickBench Q22 plan change above is the visible example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573825307
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5573802227-2206-qsxz8 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5574092146
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1179.34 ms β 1536.40 ms β 1.30x slower β
β QQuery 1 β 194.86 ms β 194.66 ms β no change β
β QQuery 2 β 487.71 ms β 489.31 ms β no change β
β QQuery 3 β 340.23 ms β 339.80 ms β no change β
β QQuery 4 β 1617.08 ms β 1689.44 ms β no change β
β QQuery 5 β 23445.42 ms β 23231.56 ms β no change β
β QQuery 6 β 4.07 ms β 4.11 ms β no change β
β QQuery 7 β 1301.43 ms β 1483.45 ms β 1.14x slower β
β QQuery 8 β 555.36 ms β 628.14 ms β 1.13x slower β
β QQuery 9 β 4175.42 ms β 4240.28 ms β no change β
β QQuery 10 β 845.14 ms β 835.28 ms β no change β
β QQuery 11 β 3022.08 ms β 2879.16 ms β no change β
β QQuery 12 β 258.91 ms β 259.58 ms β no change β
β QQuery 13 β 719.32 ms β 702.01 ms β no change β
β QQuery 14 β 3909.36 ms β 1164.88 ms β +3.36x faster β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 42055.72ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 39678.05ms β
β Average Time (HEAD) β 2803.71ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 2645.20ms β
β Queries Faster β 1 β
β Queries Slower β 3 β
β Queries with No Change β 11 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1179.34 / 1430.53 Β±130.10 / 1541.11 ms β 1536.40 / 1601.36 Β±41.80 / 1657.47 ms β 1.12x slower β
β QQuery 1 β 194.86 / 195.95 Β±1.40 / 198.72 ms β 194.66 / 196.06 Β±1.08 / 197.47 ms β no change β
β QQuery 2 β 487.71 / 494.17 Β±4.57 / 501.83 ms β 489.31 / 496.79 Β±8.86 / 510.46 ms β no change β
β QQuery 3 β 340.23 / 344.04 Β±3.37 / 348.61 ms β 339.80 / 344.13 Β±3.61 / 349.93 ms β no change β
β QQuery 4 β 1617.08 / 1863.00 Β±197.47 / 2138.51 ms β 1689.44 / 1984.97 Β±206.84 / 2218.09 ms β 1.07x slower β
β QQuery 5 β 23445.42 / 24439.55 Β±657.49 / 25419.58 ms β 23231.56 / 24375.07 Β±675.06 / 24968.54 ms β no change β
β QQuery 6 β 4.07 / 4.30 Β±0.25 / 4.77 ms β 4.11 / 5.26 Β±2.04 / 9.35 ms β 1.22x slower β
β QQuery 7 β 1301.43 / 1462.16 Β±103.00 / 1613.75 ms β 1483.45 / 1614.90 Β±133.45 / 1843.04 ms β 1.10x slower β
β QQuery 8 β 555.36 / 666.23 Β±110.82 / 879.48 ms β 628.14 / 639.60 Β±13.47 / 665.54 ms β no change β
β QQuery 9 β 4175.42 / 4359.95 Β±127.60 / 4517.43 ms β 4240.28 / 4352.07 Β±92.13 / 4463.23 ms β no change β
β QQuery 10 β 845.14 / 986.59 Β±242.37 / 1470.87 ms β 835.28 / 956.96 Β±135.44 / 1216.93 ms β no change β
β QQuery 11 β 3022.08 / 3080.84 Β±43.01 / 3138.75 ms β 2879.16 / 2988.64 Β±70.28 / 3087.70 ms β no change β
β QQuery 12 β 258.91 / 312.18 Β±80.72 / 471.34 ms β 259.58 / 272.10 Β±7.82 / 283.75 ms β +1.15x faster β
β QQuery 13 β 719.32 / 766.69 Β±57.02 / 867.70 ms β 702.01 / 753.42 Β±45.98 / 815.12 ms β no change β
β QQuery 14 β 3909.36 / 4390.31 Β±265.94 / 4696.26 ms β 1164.88 / 1293.45 Β±81.92 / 1374.22 ms β +3.39x faster β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 44796.49ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 41874.80ms β
β Average Time (HEAD) β 2986.43ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 2791.65ms β
β Queries Faster β 2 β
β Queries Slower β 4 β
β Queries with No Change β 9 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `16ace4f (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_extended`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 875.5 MiB | 873.5 MiB | -0.2% |
| Query 1 | 2.4 MiB | 2.4 MiB | +0.0% |
| Query 2 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 3 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 4 | 4.2 GiB | 4.2 GiB | +0.0% |
| Query 5 | 5.5 GiB | 5.5 GiB | +0.1% |
| Query 6 | 104 B | 104 B | +0.0% |
| Query 7 | 4.9 GiB | 4.9 GiB | +0.1% |
| Query 8 | 40.2 MiB | 40.2 MiB | +0.0% |
| Query 9 | 4.7 GiB | 4.7 GiB | -0.7% |
| Query 10 | 2.5 MiB | 2.1 MiB | -16.7% |
| Query 11 | 2.8 GiB | 2.9 GiB | +5.0% |
| Query 12 | 1.4 MiB | 1.1 MiB | -19.4% |
| Query 13 | 520 B | 520 B | +0.0% |
| Query 14 | 4.4 GiB | 1.3 GiB | -70.3% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_extended | base (`16ace4f (merge-base)`) | 5.5 GiB | 24.0 GiB | 18.5 GiB | 4.4Γ |
| clickbench_extended | changed (`claude/single-distinct-to-groupby-allow-count`) | 5.5 GiB | 23.8 GiB | 18.3 GiB | 4.3Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 230.1s |
| Peak memory | 24.0 GiB |
| Avg memory | 5.3 GiB |
| CPU user | 1989.6s |
| CPU sys | 202.0s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 215.0s |
| Peak memory | 23.8 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1911.0s |
| CPU sys | 186.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5574030670
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 794.65 ms β 794.53 ms β no change β
β QQuery 1 β 190.93 ms β 188.99 ms β no change β
β QQuery 2 β 454.77 ms β 450.98 ms β no change β
β QQuery 3 β 317.74 ms β 315.45 ms β no change β
β QQuery 4 β 1135.20 ms β 1123.91 ms β no change β
β QQuery 5 β 18139.02 ms β 18512.60 ms β no change β
β QQuery 6 β 2.63 ms β 2.65 ms β no change β
β QQuery 7 β 653.73 ms β 679.09 ms β no change β
β QQuery 8 β 417.15 ms β 417.62 ms β no change β
β QQuery 9 β 2680.54 ms β 2784.34 ms β no change β
β QQuery 10 β 649.65 ms β 646.44 ms β no change β
β QQuery 11 β 2314.52 ms β 2264.08 ms β no change β
β QQuery 12 β 187.16 ms β 190.06 ms β no change β
β QQuery 13 β 550.94 ms β 545.39 ms β no change β
β QQuery 14 β 2533.55 ms β 575.33 ms β +4.40x faster β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 31022.18ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 29491.46ms β
β Average Time (HEAD) β 2068.15ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1966.10ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 14 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 794.65 / 805.69 Β±11.35 / 826.82 ms β 794.53 / 806.07 Β±11.58 / 826.90 ms β no change β
β QQuery 1 β 190.93 / 194.26 Β±5.15 / 204.52 ms β 188.99 / 189.80 Β±0.58 / 190.59 ms β no change β
β QQuery 2 β 454.77 / 460.02 Β±2.91 / 462.78 ms β 450.98 / 454.83 Β±6.58 / 467.89 ms β no change β
β QQuery 3 β 317.74 / 319.55 Β±0.92 / 320.25 ms β 315.45 / 316.62 Β±0.83 / 317.87 ms β no change β
β QQuery 4 β 1135.20 / 1156.41 Β±15.98 / 1181.15 ms β 1123.91 / 1157.15 Β±27.73 / 1188.48 ms β no change β
β QQuery 5 β 18139.02 / 18693.77 Β±458.61 / 19387.58 ms β 18512.60 / 18799.19 Β±287.20 / 19211.52 ms β no change β
β QQuery 6 β 2.63 / 2.92 Β±0.35 / 3.58 ms β 2.65 / 2.90 Β±0.25 / 3.37 ms β no change β
β QQuery 7 β 653.73 / 708.65 Β±36.13 / 760.90 ms β 679.09 / 734.37 Β±35.46 / 774.99 ms β no change β
β QQuery 8 β 417.15 / 437.23 Β±30.45 / 497.78 ms β 417.62 / 454.35 Β±62.10 / 578.36 ms β no change β
β QQuery 9 β 2680.54 / 2905.35 Β±117.53 / 3013.57 ms β 2784.34 / 2843.09 Β±48.25 / 2918.64 ms β no change β
β QQuery 10 β 649.65 / 669.40 Β±19.36 / 695.36 ms β 646.44 / 666.95 Β±15.96 / 688.53 ms β no change β
β QQuery 11 β 2314.52 / 2396.04 Β±41.81 / 2428.78 ms β 2264.08 / 2300.30 Β±32.20 / 2345.94 ms β no change β
β QQuery 12 β 187.16 / 212.93 Β±42.76 / 298.27 ms β 190.06 / 221.10 Β±52.54 / 325.88 ms β no change β
β QQuery 13 β 550.94 / 570.80 Β±18.45 / 600.06 ms β 545.39 / 582.66 Β±36.38 / 648.87 ms β no change β
β QQuery 14 β 2533.55 / 2588.73 Β±44.49 / 2666.85 ms β 575.33 / 624.47 Β±47.71 / 714.36 ms β +4.15x faster β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 32121.74ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 30153.86ms β
β Average Time (HEAD) β 2141.45ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 2010.26ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 14 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `16ace4f (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_extended`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 865.0 MiB | 855.0 MiB | -1.2% |
| Query 1 | 2.4 MiB | 2.4 MiB | +0.0% |
| Query 2 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 3 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 4 | 4.2 GiB | 4.2 GiB | +0.0% |
| Query 5 | 5.5 GiB | 5.5 GiB | +0.1% |
| Query 6 | 104 B | 104 B | +0.0% |
| Query 7 | 4.9 GiB | 4.9 GiB | -0.0% |
| Query 8 | 38.7 MiB | 36.9 MiB | -4.8% |
| Query 9 | 4.6 GiB | 4.6 GiB | -0.2% |
| Query 10 | 2.1 MiB | 2.1 MiB | +0.5% |
| Query 11 | 2.8 GiB | 2.9 GiB | +2.3% |
| Query 12 | 1.4 MiB | 1.4 MiB | +0.4% |
| Query 13 | 520 B | 520 B | +0.0% |
| Query 14 | 4.4 GiB | 1.3 GiB | -69.7% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_extended | base (`16ace4f (merge-base)`) | 5.5 GiB | 24.1 GiB | 18.6 GiB | 4.4Γ |
| clickbench_extended | changed (`claude/single-distinct-to-groupby-allow-count`) | 5.5 GiB | 24.1 GiB | 18.6 GiB | 4.4Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 165.0s |
| Peak memory | 24.1 GiB |
| Avg memory | 5.3 GiB |
| CPU user | 1521.2s |
| CPU sys | 91.8s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 24.1 GiB |
| Avg memory | 5.2 GiB |
| CPU user | 1443.3s |
| CPU sys | 89.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5586187503
Thanks for the review @kosiew ππ»
I addressed your feedback but in poking around I decided this needs one more pass. I'll bother you with another request for review once complete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5567074926
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 871.78 ms β 842.28 ms β no change β
β QQuery 1 β 189.24 ms β 189.30 ms β no change β
β QQuery 2 β 452.30 ms β 453.42 ms β no change β
β QQuery 3 β 315.07 ms β 318.76 ms β no change β
β QQuery 4 β 1205.25 ms β 1177.92 ms β no change β
β QQuery 5 β 11091.47 ms β 11965.78 ms β 1.08x slower β
β QQuery 6 β 2.81 ms β 2.80 ms β no change β
β QQuery 7 β 736.12 ms β 732.20 ms β no change β
β QQuery 8 β 428.79 ms β 434.64 ms β no change β
β QQuery 9 β 2983.43 ms β 3273.37 ms β 1.10x slower β
β QQuery 10 β 642.03 ms β 644.94 ms β no change β
β QQuery 11 β 2374.74 ms β 2196.24 ms β +1.08x faster β
β QQuery 12 β 188.37 ms β 193.09 ms β no change β
β QQuery 13 β 544.45 ms β 561.91 ms β no change β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 22025.85ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 22986.66ms β
β Average Time (HEAD) β 1573.27ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1641.90ms β
β Queries Faster β 1 β
β Queries Slower β 2 β
β Queries with No Change β 11 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 871.78 / 899.98 Β±30.52 / 956.01 ms β 842.28 / 891.77 Β±34.44 / 926.23 ms β no change β
β QQuery 1 β 189.24 / 190.70 Β±2.04 / 194.69 ms β 189.30 / 192.50 Β±5.38 / 203.23 ms β no change β
β QQuery 2 β 452.30 / 455.99 Β±2.11 / 458.54 ms β 453.42 / 457.10 Β±2.50 / 460.22 ms β no change β
β QQuery 3 β 315.07 / 317.98 Β±1.72 / 319.96 ms β 318.76 / 319.67 Β±1.08 / 321.78 ms β no change β
β QQuery 4 β 1205.25 / 1215.29 Β±6.71 / 1223.42 ms β 1177.92 / 1205.89 Β±14.67 / 1218.42 ms β no change β
β QQuery 5 β 11091.47 / 11751.80 Β±341.61 / 12017.22 ms β 11965.78 / 12053.71 Β±101.56 / 12239.90 ms β no change β
β QQuery 6 β 2.81 / 3.01 Β±0.30 / 3.61 ms β 2.80 / 3.14 Β±0.39 / 3.90 ms β no change β
β QQuery 7 β 736.12 / 817.21 Β±68.39 / 932.37 ms β 732.20 / 800.69 Β±61.22 / 911.68 ms β no change β
β QQuery 8 β 428.79 / 455.92 Β±46.04 / 547.85 ms β 434.64 / 447.84 Β±16.72 / 480.57 ms β no change β
β QQuery 9 β 2983.43 / 3053.13 Β±49.69 / 3117.18 ms β 3273.37 / 3362.39 Β±75.13 / 3494.95 ms β 1.10x slower β
β QQuery 10 β 642.03 / 702.43 Β±102.01 / 905.62 ms β 644.94 / 709.90 Β±97.38 / 902.42 ms β no change β
β QQuery 11 β 2374.74 / 2430.13 Β±35.60 / 2476.24 ms β 2196.24 / 2251.10 Β±44.17 / 2312.21 ms β +1.08x faster β
β QQuery 12 β 188.37 / 196.72 Β±9.73 / 214.20 ms β 193.09 / 218.42 Β±37.31 / 292.58 ms β 1.11x slower β
β QQuery 13 β 544.45 / 559.90 Β±11.33 / 571.47 ms β 561.91 / 583.92 Β±26.50 / 634.54 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 23050.20ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 23498.04ms β
β Average Time (HEAD) β 1646.44ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1678.43ms β
β Queries Faster β 1 β
β Queries Slower β 2 β
β Queries with No Change β 11 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 120.0s |
| Peak memory | 11.4 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1065.8s |
| CPU sys | 50.4s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 120.0s |
| Peak memory | 11.1 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 1063.1s |
| CPU sys | 51.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5572016976
@kosiew breaking out the benchmark in https://github.com/apache/datafusion/pull/25026 + a fix to our benchmarks in https://github.com/apache/datafusion/pull/25027 that I found as a drive by
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5566637455
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5566611080-2195-sk5cw 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5566661958
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5566611080-2197-5pnzw 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5515057192
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.25 ms β 1.26 ms β no change β
β QQuery 1 β 12.04 ms β 12.31 ms β no change β
β QQuery 2 β 37.24 ms β 36.99 ms β no change β
β QQuery 3 β 32.08 ms β 31.95 ms β no change β
β QQuery 4 β 238.55 ms β 234.46 ms β no change β
β QQuery 5 β 283.29 ms β 283.96 ms β no change β
β QQuery 6 β 1.29 ms β 1.33 ms β no change β
β QQuery 7 β 13.56 ms β 13.59 ms β no change β
β QQuery 8 β 359.70 ms β 359.76 ms β no change β
β QQuery 9 β 484.05 ms β 496.72 ms β no change β
β QQuery 10 β 72.00 ms β 73.83 ms β no change β
β QQuery 11 β 83.33 ms β 84.50 ms β no change β
β QQuery 12 β 280.11 ms β 285.04 ms β no change β
β QQuery 13 β 1018.57 ms β 1021.09 ms β no change β
β QQuery 14 β 296.70 ms β 296.85 ms β no change β
β QQuery 15 β 286.15 ms β 291.21 ms β no change β
β QQuery 16 β 1238.77 ms β 1238.03 ms β no change β
β QQuery 17 β 958.92 ms β 999.14 ms β no change β
β QQuery 18 β 2558.25 ms β 2624.19 ms β no change β
β QQuery 19 β 29.59 ms β 29.33 ms β no change β
β QQuery 20 β 517.42 ms β 529.51 ms β no change β
β QQuery 21 β 516.30 ms β 527.07 ms β no change β
β QQuery 22 β 1002.16 ms β 1014.28 ms β no change β
β QQuery 23 β 3089.00 ms β 3127.31 ms β no change β
β QQuery 24 β 42.94 ms β 41.81 ms β no change β
β QQuery 25 β 113.80 ms β 113.77 ms β no change β
β QQuery 26 β 43.30 ms β 42.36 ms β no change β
β QQuery 27 β 518.63 ms β 524.14 ms β no change β
β QQuery 28 β 2978.05 ms β 2986.05 ms β no change β
β QQuery 29 β 42.11 ms β 42.46 ms β no change β
β QQuery 30 β 327.55 ms β 321.58 ms β no change β
β QQuery 31 β 291.20 ms β 293.10 ms β no change β
β QQuery 32 β 3488.76 ms β 3437.55 ms β no change β
β QQuery 33 β 2680.67 ms β 2632.40 ms β no change β
β QQuery 34 β 2798.53 ms β 2759.08 ms β no change β
β QQuery 35 β 310.34 ms β 305.11 ms β no change β
β QQuery 36 β 67.79 ms β 68.55 ms β no change β
β QQuery 37 β 37.27 ms β 36.66 ms β no change β
β QQuery 38 β 41.42 ms β 41.20 ms β no change β
β QQuery 39 β 136.99 ms β 137.55 ms β no change β
β QQuery 40 β 15.21 ms β 15.10 ms β no change β
β QQuery 41 β 14.91 ms β 14.67 ms β no change β
β QQuery 42 β 14.52 ms β 14.10 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 27374.33ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 27440.98ms β
β Average Time (HEAD) β 636.61ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 638.16ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 43 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.25 / 4.13 Β±5.61 / 15.36 ms β 1.26 / 4.21 Β±5.70 / 15.60 ms β no change β
β QQuery 1 β 12.04 / 12.53 Β±0.27 / 12.81 ms β 12.31 / 12.58 Β±0.22 / 12.93 ms β no change β
β QQuery 2 β 37.24 / 37.77 Β±0.36 / 38.15 ms β 36.99 / 37.28 Β±0.33 / 37.91 ms β no change β
β QQuery 3 β 32.08 / 33.20 Β±0.78 / 34.13 ms β 31.95 / 32.11 Β±0.13 / 32.28 ms β no change β
β QQuery 4 β 238.55 / 241.74 Β±2.31 / 245.36 ms β 234.46 / 241.16 Β±4.40 / 247.32 ms β no change β
β QQuery 5 β 283.29 / 287.49 Β±3.02 / 291.53 ms β 283.96 / 290.74 Β±4.13 / 296.74 ms β no change β
β QQuery 6 β 1.29 / 1.43 Β±0.22 / 1.87 ms β 1.33 / 1.49 Β±0.24 / 1.96 ms β no change β
β QQuery 7 β 13.56 / 14.48 Β±1.15 / 16.75 ms β 13.59 / 15.23 Β±2.48 / 20.17 ms β 1.05x slower β
β QQuery 8 β 359.70 / 364.02 Β±2.84 / 366.95 ms β 359.76 / 364.52 Β±2.98 / 369.14 ms β no change β
β QQuery 9 β 484.05 / 495.25 Β±7.45 / 504.76 ms β 496.72 / 504.66 Β±8.58 / 520.61 ms β no change β
β QQuery 10 β 72.00 / 72.65 Β±0.49 / 73.38 ms β 73.83 / 75.06 Β±0.97 / 76.47 ms β no change β
β QQuery 11 β 83.33 / 84.08 Β±0.90 / 85.80 ms β 84.50 / 87.00 Β±2.99 / 92.69 ms β no change β
β QQuery 12 β 280.11 / 286.46 Β±4.35 / 292.04 ms β 285.04 / 300.48 Β±12.85 / 320.77 ms β no change β
β QQuery 13 β 1018.57 / 1028.17 Β±13.05 / 1054.04 ms β 1021.09 / 1028.06 Β±3.86 / 1032.88 ms β no change β
β QQuery 14 β 296.70 / 302.18 Β±4.52 / 307.42 ms β 296.85 / 309.11 Β±18.89 / 346.57 ms β no change β
β QQuery 15 β 286.15 / 296.29 Β±14.52 / 324.69 ms β 291.21 / 298.52 Β±9.86 / 317.77 ms β no change β
β QQuery 16 β 1238.77 / 1291.13 Β±34.65 / 1338.84 ms β 1238.03 / 1326.62 Β±74.17 / 1458.76 ms β no change β
β QQuery 17 β 958.92 / 981.52 Β±24.26 / 1026.10 ms β 999.14 / 1011.04 Β±19.32 / 1049.47 ms β no change β
β QQuery 18 β 2558.25 / 2607.84 Β±28.90 / 2637.26 ms β 2624.19 / 2684.03 Β±36.03 / 2735.05 ms β no change β
β QQuery 19 β 29.59 / 34.34 Β±7.12 / 48.36 ms β 29.33 / 29.71 Β±0.44 / 30.52 ms β +1.16x faster β
β QQuery 20 β 517.42 / 534.52 Β±10.93 / 550.84 ms β 529.51 / 534.55 Β±4.05 / 540.57 ms β no change β
β QQuery 21 β 516.30 / 526.22 Β±7.42 / 536.04 ms β 527.07 / 534.47 Β±5.75 / 544.70 ms β no change β
β QQuery 22 β 1002.16 / 1004.25 Β±2.64 / 1009.41 ms β 1014.28 / 1022.86 Β±6.46 / 1032.03 ms β no change β
β QQuery 23 β 3089.00 / 3168.46 Β±48.27 / 3237.70 ms β 3127.31 / 3152.36 Β±15.67 / 3172.62 ms β no change β
β QQuery 24 β 42.94 / 43.79 Β±0.86 / 45.43 ms β 41.81 / 46.31 Β±6.05 / 57.74 ms β 1.06x slower β
β QQuery 25 β 113.80 / 115.70 Β±1.56 / 118.50 ms β 113.77 / 115.43 Β±1.50 / 117.97 ms β no change β
β QQuery 26 β 43.30 / 47.76 Β±4.32 / 54.98 ms β 42.36 / 44.15 Β±1.81 / 47.37 ms β +1.08x faster β
β QQuery 27 β 518.63 / 527.15 Β±5.77 / 534.51 ms β 524.14 / 531.48 Β±7.46 / 543.78 ms β no change β
β QQuery 28 β 2978.05 / 3085.15 Β±76.47 / 3198.31 ms β 2986.05 / 3002.24 Β±14.17 / 3023.59 ms β no change β
β QQuery 29 β 42.11 / 43.05 Β±0.84 / 44.28 ms β 42.46 / 42.93 Β±0.40 / 43.59 ms β no change β
β QQuery 30 β 327.55 / 334.22 Β±4.62 / 341.41 ms β 321.58 / 335.40 Β±11.36 / 351.21 ms β no change β
β QQuery 31 β 291.20 / 305.89 Β±10.06 / 321.25 ms β 293.10 / 300.07 Β±4.92 / 307.82 ms β no change β
β QQuery 32 β 3488.76 / 3514.56 Β±16.08 / 3536.17 ms β 3437.55 / 3484.28 Β±33.12 / 3524.84 ms β no change β
β QQuery 33 β 2680.67 / 2854.69 Β±114.70 / 3012.87 ms β 2632.40 / 2720.12 Β±76.93 / 2839.08 ms β no change β
β QQuery 34 β 2798.53 / 2899.67 Β±58.26 / 2964.30 ms β 2759.08 / 2795.77 Β±25.02 / 2828.75 ms β no change β
β QQuery 35 β 310.34 / 316.23 Β±5.72 / 325.13 ms β 305.11 / 321.58 Β±9.58 / 332.81 ms β no change β
β QQuery 36 β 67.79 / 73.23 Β±6.64 / 85.89 ms β 68.55 / 73.16 Β±3.75 / 78.74 ms β no change β
β QQuery 37 β 37.27 / 42.84 Β±8.99 / 60.50 ms β 36.66 / 49.52 Β±24.98 / 99.48 ms β 1.16x slower β
β QQuery 38 β 41.42 / 43.62 Β±2.67 / 47.65 ms β 41.20 / 41.59 Β±0.23 / 41.84 ms β no change β
β QQuery 39 β 136.99 / 150.38 Β±11.56 / 170.92 ms β 137.55 / 155.87 Β±12.92 / 168.88 ms β no change β
β QQuery 40 β 15.21 / 15.81 Β±0.61 / 16.88 ms β 15.10 / 15.52 Β±0.22 / 15.68 ms β no change β
β QQuery 41 β 14.91 / 15.22 Β±0.29 / 15.62 ms β 14.67 / 14.86 Β±0.17 / 15.18 ms β no change β
β QQuery 42 β 14.52 / 14.57 Β±0.05 / 14.63 ms β 14.10 / 14.42 Β±0.19 / 14.65 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 28153.72ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 28002.54ms β
β Average Time (HEAD) β 654.74ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 651.22ms β
β Queries Faster β 2 β
β Queries Slower β 3 β
β Queries with No Change β 38 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `da89c7c (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 777.8 MiB | 770.4 MiB | -1.0% |
| Query 5 | 1.2 GiB | 1.2 GiB | +2.3% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 40.1 MiB | 60.1 MiB | +49.8% |
| Query 8 | 849.5 MiB | 849.1 MiB | -0.0% |
| Query 9 | 593.7 MiB | 593.9 MiB | +0.0% |
| Query 10 | 114.7 MiB | 111.8 MiB | -2.5% |
| Query 11 | 107.8 MiB | 112.1 MiB | +4.0% |
| Query 12 | 1.3 GiB | 1.3 GiB | +5.2% |
| Query 13 | 1.0 GiB | 1.0 GiB | -0.7% |
| Query 14 | 1.3 GiB | 1.3 GiB | +1.5% |
| Query 15 | 1.2 GiB | 1.2 GiB | -1.6% |
| Query 16 | 1.8 GiB | 2.1 GiB | +16.7% |
| Query 17 | 1.9 GiB | 2.0 GiB | +6.9% |
| Query 18 | 1.9 GiB | 1.8 GiB | -5.8% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.6 MiB | +9.3% |
| Query 22 | 4.8 MiB | 3.0 MiB | -37.3% |
| Query 23 | 26.8 MiB | 31.9 MiB | +19.0% |
| Query 24 | 59.3 MiB | 59.1 MiB | -0.3% |
| Query 25 | 176.1 MiB | 184.0 MiB | +4.5% |
| Query 26 | 64.8 MiB | 60.9 MiB | -5.9% |
| Query 27 | 2.2 MiB | 2.4 MiB | +10.0% |
| Query 28 | 1.5 GiB | 1.5 GiB | -1.2% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 716.0 MiB | 697.8 MiB | -2.5% |
| Query 31 | 1.5 GiB | 1.5 GiB | +1.6% |
| Query 32 | 970.2 MiB | 928.1 MiB | -4.3% |
| Query 33 | 2.1 GiB | 2.1 GiB | +1.2% |
| Query 34 | 2.1 GiB | 2.1 GiB | +1.3% |
| Query 35 | 602.6 MiB | 608.0 MiB | +0.9% |
| Query 36 | 123.2 MiB | 118.1 MiB | -4.2% |
| Query 37 | 6.3 MiB | 7.5 MiB | +20.0% |
| Query 38 | 5.2 MiB | 4.7 MiB | -10.6% |
| Query 39 | 298.1 MiB | 298.1 MiB | +0.0% |
| Query 40 | 2.0 MiB | 2.0 MiB | -0.1% |
| Query 41 | 3.1 MiB | 3.1 MiB | +0.0% |
| Query 42 | 1.7 MiB | 2.1 MiB | +24.5% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`da89c7c (merge-base)`) | 2.1 GiB | 8.4 GiB | 6.3 GiB | 4.0Γ |
| clickbench_partitioned | changed (`claude/single-distinct-to-groupby-allow-count`) | 2.1 GiB | 8.4 GiB | 6.3 GiB | 4.0Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 145.0s |
| Peak memory | 8.4 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1430.0s |
| CPU sys | 136.8s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 145.0s |
| Peak memory | 8.4 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1420.1s |
| CPU sys | 139.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5514791127
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5514755290-2104-xv7bw 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5585939361
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5585614783)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (173ae06366a11599f7f443382f7b27ca57854df2) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..173ae06366a11599f7f443382f7b27ca57854df2)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 ms β 1.25 ms β no change β
β QQuery 1 β 11.68 ms β 12.03 ms β no change β
β QQuery 2 β 37.50 ms β 37.32 ms β no change β
β QQuery 3 β 31.02 ms β 31.03 ms β no change β
β QQuery 4 β 221.19 ms β 221.65 ms β no change β
β QQuery 5 β 269.59 ms β 269.37 ms β no change β
β QQuery 6 β 1.30 ms β 1.28 ms β no change β
β QQuery 7 β 12.86 ms β 13.30 ms β no change β
β QQuery 8 β 326.34 ms β 329.13 ms β no change β
β QQuery 9 β 451.18 ms β 450.49 ms β no change β
β QQuery 10 β 68.53 ms β 69.50 ms β no change β
β QQuery 11 β 79.99 ms β 80.01 ms β no change β
β QQuery 12 β 265.08 ms β 266.12 ms β no change β
β QQuery 13 β 956.19 ms β 957.06 ms β no change β
β QQuery 14 β 280.88 ms β 279.41 ms β no change β
β QQuery 15 β 261.79 ms β 262.36 ms β no change β
β QQuery 16 β 1200.53 ms β 1218.74 ms β no change β
β QQuery 17 β 928.57 ms β 904.87 ms β no change β
β QQuery 18 β 2434.58 ms β 2465.65 ms β no change β
β QQuery 19 β 27.37 ms β 27.64 ms β no change β
β QQuery 20 β 515.87 ms β 517.77 ms β no change β
β QQuery 21 β 513.02 ms β 513.55 ms β no change β
β QQuery 22 β 975.75 ms β 976.75 ms β no change β
β QQuery 23 β 2960.84 ms β 2978.08 ms β no change β
β QQuery 24 β 40.53 ms β 40.86 ms β no change β
β QQuery 25 β 110.21 ms β 109.47 ms β no change β
β QQuery 26 β 40.94 ms β 41.05 ms β no change β
β QQuery 27 β 512.39 ms β 512.67 ms β no change β
β QQuery 28 β 2904.32 ms β 2922.14 ms β no change β
β QQuery 29 β 41.69 ms β 41.44 ms β no change β
β QQuery 30 β 302.95 ms β 298.48 ms β no change β
β QQuery 31 β 279.90 ms β 277.81 ms β no change β
β QQuery 32 β 3225.83 ms β 3228.48 ms β no change β
β QQuery 33 β 2564.89 ms β 2541.93 ms β no change β
β QQuery 34 β 2536.30 ms β 2587.22 ms β no change β
β QQuery 35 β 275.84 ms β 272.24 ms β no change β
β QQuery 36 β 65.71 ms β 67.01 ms β no change β
β QQuery 37 β 35.33 ms β 35.30 ms β no change β
β QQuery 38 β 39.73 ms β 40.79 ms β no change β
β QQuery 39 β 135.45 ms β 131.43 ms β no change β
β QQuery 40 β 13.92 ms β 14.14 ms β no change β
β QQuery 41 β 13.51 ms β 13.17 ms β no change β
β QQuery 42 β 12.99 ms β 12.92 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 25985.29ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 26072.93ms β
β Average Time (HEAD) β 604.31ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 606.35ms β
β Queries Faster β 0 β
β Queries Slower β 0 β
β Queries with No Change β 43 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 / 3.91 Β±5.33 / 14.58 ms β 1.25 / 3.97 Β±5.37 / 14.71 ms β no change β
β QQuery 1 β 11.68 / 11.94 Β±0.14 / 12.07 ms β 12.03 / 12.32 Β±0.22 / 12.69 ms β no change β
β QQuery 2 β 37.50 / 37.77 Β±0.24 / 38.17 ms β 37.32 / 37.75 Β±0.63 / 38.98 ms β no change β
β QQuery 3 β 31.02 / 31.53 Β±0.73 / 32.98 ms β 31.03 / 31.31 Β±0.28 / 31.77 ms β no change β
β QQuery 4 β 221.19 / 224.04 Β±2.27 / 227.93 ms β 221.65 / 223.46 Β±1.17 / 224.90 ms β no change β
β QQuery 5 β 269.59 / 271.49 Β±1.18 / 273.20 ms β 269.37 / 272.36 Β±1.94 / 274.88 ms β no change β
β QQuery 6 β 1.30 / 1.44 Β±0.22 / 1.87 ms β 1.28 / 1.43 Β±0.21 / 1.84 ms β no change β
β QQuery 7 β 12.86 / 13.11 Β±0.17 / 13.37 ms β 13.30 / 13.37 Β±0.07 / 13.46 ms β no change β
β QQuery 8 β 326.34 / 332.09 Β±4.45 / 339.84 ms β 329.13 / 333.91 Β±3.92 / 339.24 ms β no change β
β QQuery 9 β 451.18 / 457.68 Β±4.16 / 463.55 ms β 450.49 / 455.52 Β±3.60 / 460.05 ms β no change β
β QQuery 10 β 68.53 / 71.31 Β±4.52 / 80.34 ms β 69.50 / 72.23 Β±4.12 / 80.40 ms β no change β
β QQuery 11 β 79.99 / 80.58 Β±0.31 / 80.85 ms β 80.01 / 81.13 Β±0.76 / 82.19 ms β no change β
β QQuery 12 β 265.08 / 269.64 Β±3.57 / 275.49 ms β 266.12 / 272.46 Β±7.18 / 283.77 ms β no change β
β QQuery 13 β 956.19 / 969.53 Β±10.63 / 984.85 ms β 957.06 / 976.17 Β±12.86 / 996.44 ms β no change β
β QQuery 14 β 280.88 / 283.62 Β±3.21 / 289.84 ms β 279.41 / 283.63 Β±2.29 / 285.99 ms β no change β
β QQuery 15 β 261.79 / 269.84 Β±5.48 / 275.99 ms β 262.36 / 267.68 Β±4.38 / 274.46 ms β no change β
β QQuery 16 β 1200.53 / 1219.68 Β±18.42 / 1248.10 ms β 1218.74 / 1247.92 Β±21.47 / 1278.46 ms β no change β
β QQuery 17 β 928.57 / 951.46 Β±18.07 / 972.90 ms β 904.87 / 914.21 Β±7.34 / 922.41 ms β no change β
β QQuery 18 β 2434.58 / 2476.29 Β±36.51 / 2542.58 ms β 2465.65 / 2486.71 Β±29.59 / 2542.42 ms β no change β
β QQuery 19 β 27.37 / 31.96 Β±5.58 / 39.84 ms β 27.64 / 28.73 Β±1.31 / 30.65 ms β +1.11x faster β
β QQuery 20 β 515.87 / 525.39 Β±8.32 / 536.87 ms β 517.77 / 521.42 Β±3.40 / 527.72 ms β no change β
β QQuery 21 β 513.02 / 522.00 Β±7.04 / 532.59 ms β 513.55 / 519.13 Β±5.19 / 528.96 ms β no change β
β QQuery 22 β 975.75 / 985.09 Β±9.50 / 1003.37 ms β 976.75 / 981.07 Β±3.90 / 987.00 ms β no change β
β QQuery 23 β 2960.84 / 2982.94 Β±19.51 / 3011.99 ms β 2978.08 / 3039.42 Β±36.09 / 3085.69 ms β no change β
β QQuery 24 β 40.53 / 49.60 Β±17.64 / 84.87 ms β 40.86 / 41.67 Β±0.69 / 42.72 ms β +1.19x faster β
β QQuery 25 β 110.21 / 116.62 Β±10.09 / 136.27 ms β 109.47 / 111.82 Β±2.20 / 115.88 ms β no change β
β QQuery 26 β 40.94 / 42.16 Β±1.05 / 43.98 ms β 41.05 / 41.25 Β±0.21 / 41.61 ms β no change β
β QQuery 27 β 512.39 / 515.08 Β±3.87 / 522.74 ms β 512.67 / 528.07 Β±15.68 / 558.10 ms β no change β
β QQuery 28 β 2904.32 / 2932.17 Β±14.95 / 2943.95 ms β 2922.14 / 2932.76 Β±7.54 / 2944.08 ms β no change β
β QQuery 29 β 41.69 / 57.91 Β±24.17 / 104.36 ms β 41.44 / 43.49 Β±3.47 / 50.40 ms β +1.33x faster β
β QQuery 30 β 302.95 / 314.95 Β±12.52 / 335.35 ms β 298.48 / 303.62 Β±3.87 / 308.45 ms β no change β
β QQuery 31 β 279.90 / 284.47 Β±4.57 / 292.33 ms β 277.81 / 296.29 Β±26.77 / 349.15 ms β no change β
β QQuery 32 β 3225.83 / 3272.59 Β±31.22 / 3309.86 ms β 3228.48 / 3278.90 Β±50.47 / 3363.90 ms β no change β
β QQuery 33 β 2564.89 / 2596.76 Β±27.27 / 2641.47 ms β 2541.93 / 2616.73 Β±60.72 / 2709.78 ms β no change β
β QQuery 34 β 2536.30 / 2623.74 Β±51.75 / 2677.82 ms β 2587.22 / 2628.46 Β±36.65 / 2675.04 ms β no change β
β QQuery 35 β 275.84 / 281.97 Β±4.57 / 286.80 ms β 272.24 / 285.56 Β±10.47 / 298.87 ms β no change β
β QQuery 36 β 65.71 / 71.08 Β±4.26 / 76.24 ms β 67.01 / 69.29 Β±1.83 / 71.04 ms β no change β
β QQuery 37 β 35.33 / 39.44 Β±6.73 / 52.84 ms β 35.30 / 47.78 Β±21.84 / 91.36 ms β 1.21x slower β
β QQuery 38 β 39.73 / 41.71 Β±1.32 / 43.89 ms β 40.79 / 43.15 Β±1.45 / 45.00 ms β no change β
β QQuery 39 β 135.45 / 154.19 Β±16.86 / 184.94 ms β 131.43 / 139.12 Β±8.24 / 151.85 ms β +1.11x faster β
β QQuery 40 β 13.92 / 15.47 Β±2.43 / 20.29 ms β 14.14 / 14.32 Β±0.16 / 14.54 ms β +1.08x faster β
β QQuery 41 β 13.51 / 14.03 Β±0.26 / 14.23 ms β 13.17 / 14.76 Β±2.20 / 19.11 ms β 1.05x slower β
β QQuery 42 β 12.99 / 14.65 Β±2.60 / 19.82 ms β 12.92 / 13.05 Β±0.16 / 13.31 ms β +1.12x faster β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26462.94ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 26527.38ms β
β Average Time (HEAD) β 615.42ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 616.92ms β
β Queries Faster β 6 β
β Queries Slower β 2 β
β Queries with No Change β 35 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `16ace4f (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 754.3 MiB | 754.9 MiB | +0.1% |
| Query 5 | 1.2 GiB | 1.2 GiB | -2.7% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 50.2 MiB | 40.3 MiB | -19.7% |
| Query 8 | 869.2 MiB | 892.2 MiB | +2.6% |
| Query 9 | 593.7 MiB | 593.9 MiB | +0.0% |
| Query 10 | 111.9 MiB | 112.8 MiB | +0.7% |
| Query 11 | 121.7 MiB | 118.7 MiB | -2.5% |
| Query 12 | 1.4 GiB | 1.3 GiB | -1.7% |
| Query 13 | 1.0 GiB | 1.0 GiB | +1.7% |
| Query 14 | 1.3 GiB | 1.3 GiB | -0.0% |
| Query 15 | 1.1 GiB | 1.1 GiB | +2.0% |
| Query 16 | 1.8 GiB | 2.0 GiB | +11.1% |
| Query 17 | 1.8 GiB | 1.7 GiB | -2.5% |
| Query 18 | 2.1 GiB | 1.9 GiB | -6.3% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.3 MiB | +0.6% |
| Query 22 | 2.5 MiB | 3.0 MiB | +22.0% |
| Query 23 | 33.2 MiB | 24.5 MiB | -26.1% |
| Query 24 | 59.5 MiB | 58.5 MiB | -1.7% |
| Query 25 | 173.3 MiB | 174.3 MiB | +0.6% |
| Query 26 | 60.8 MiB | 61.3 MiB | +0.9% |
| Query 27 | 2.2 MiB | 2.4 MiB | +10.0% |
| Query 28 | 1.5 GiB | 1.5 GiB | -4.7% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 680.8 MiB | 671.1 MiB | -1.4% |
| Query 31 | 1.5 GiB | 1.4 GiB | -3.2% |
| Query 32 | 928.1 MiB | 929.0 MiB | +0.1% |
| Query 33 | 2.1 GiB | 2.1 GiB | -1.3% |
| Query 34 | 2.2 GiB | 2.1 GiB | -1.5% |
| Query 35 | 602.9 MiB | 624.7 MiB | +3.6% |
| Query 36 | 121.1 MiB | 113.6 MiB | -6.2% |
| Query 37 | 7.4 MiB | 6.9 MiB | -7.2% |
| Query 38 | 5.2 MiB | 5.7 MiB | +9.1% |
| Query 39 | 297.8 MiB | 298.1 MiB | +0.1% |
| Query 40 | 2.0 MiB | 2.0 MiB | -1.1% |
| Query 41 | 3.1 MiB | 3.1 MiB | +0.0% |
| Query 42 | 1.7 MiB | 1.6 MiB | -2.3% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`16ace4f (merge-base)`) | 2.2 GiB | 8.6 GiB | 6.4 GiB | 4.0Γ |
| clickbench_partitioned | changed (`claude/single-distinct-to-groupby-allow-count`) | 2.1 GiB | 9.3 GiB | 7.2 GiB | 4.4Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 135.0s |
| Peak memory | 8.6 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1342.2s |
| CPU sys | 127.6s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 135.0s |
| Peak memory | 9.3 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 1345.1s |
| CPU sys | 129.3s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5499104688
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24859?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `84.69945%` with `28 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.60%. Comparing base ([`da89c7c`](https://app.codecov.io/gh/apache/datafusion/commit/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`f47c045`](https://app.codecov.io/gh/apache/datafusion/commit/f47c04523f7f84ad0993001b47ae1bc2ce92b0d5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24859?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...fusion/optimizer/src/single\_distinct\_to\_groupby.rs](https://app.codecov.io/gh/apache/datafusion/pull/24859?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fsingle_distinct_to_groupby.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3NpbmdsZV9kaXN0aW5jdF90b19ncm91cGJ5LnJz) | 84.69% | [9 Missing and 19 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24859?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24859 +/- ##
========================================
Coverage 81.60% 81.60%
========================================
Files 1123 1123
Lines 408898 409051 +153
Branches 408898 409051 +153
========================================
+ Hits 333670 333806 +136
- Misses 55625 55627 +2
- Partials 19603 19618 +15
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24859?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#discussion_r3955646518
##########
datafusion/substrait/tests/cases/roundtrip_logical_plan.rs:
##########
@@ -333,8 +333,13 @@ async fn simple_aggregate() -> Result<()> {
#[tokio::test]
async fn aggregate_distinct_with_having() -> Result<()> {
- roundtrip("SELECT a, count(distinct b) FROM data GROUP BY a, c HAVING count(b) > 100")
- .await
+ let ctx = create_context_without_single_distinct_to_group_by().await?;
Review Comment:
I think using the custom context here makes sense. Substrait does not represent the aggregate/group-expression aliases introduced by this rewrite, so this looks like an existing plan-identity limitation rather than a semantic regression.
Would it be worth adding a separate test using the default context that exercises the rewritten plan through Substrait and checks its schema and query results instead of exact plan identity? This test could then stay focused on identity roundtripping with `single_distinct_aggregation_to_group_by` disabled.
##########
datafusion/optimizer/src/single_distinct_to_groupby.rs:
##########
@@ -96,7 +170,58 @@ fn is_single_distinct_agg(aggr_expr: &[Expr]) -> Result<bool> {
return Ok(false);
}
}
- Ok(aggregate_count == aggr_expr.len() && fields_set.len() == 1)
+ if aggregate_count != aggr_expr.len() || fields_set.len() != 1 {
+ return Ok(false);
+ }
+ if has_count_rollup && !rewrite_pays_for_count(&distinct_aggs, input_schema)? {
+ return Ok(false);
+ }
+ Ok(true)
+}
+
+/// Whether the rewrite is worth extending to a plan that only qualifies because
+/// of the non-distinct `count`.
+///
+/// The rewrite is not free: every other aggregate moves down to the inner group
+/// by, which has a row per `(group, distinct value)` pair rather than per group,
+/// and each one keeps its state at that finer grain. What pays for it is taking
+/// the distinct aggregate off `GroupsAccumulatorAdapter`, whose one boxed
+/// accumulator per group is the expensive shape. A distinct aggregate that
+/// already has a specialized `GroupsAccumulator` never went near the adapter, so
+/// there is nothing to buy and only the inner group by to pay for: ClickBench
+/// Q22, whose `count(DISTINCT "UserID")` is over an `Int64`, measured a 132%
+/// increase in peak memory pool reservation when the rewrite applied to it.
+///
+/// `Some(false)` is the only answer that buys anything. `Some(true)` says the
+/// call never reaches the adapter. `None` says the function does not answer the
+/// question from the argument types, which is the default and so the answer for
+/// almost every function. Silence is not evidence, and reading it as `Some(false)`
+/// would open this path to every such function: `sum(DISTINCT int_col)` beside a
+/// `count(*)` measured 3.15x the peak memory once rewritten, over 4,000,000 rows
+/// in 2,000 groups.
+///
+/// The predicate is a proxy, not the true discriminator. What decides the
+/// outcome is the cost per distinct value on each side, which this rule cannot
+/// see. The proxy is deliberately conservative in the direction that leaves a
+/// plan alone.
+///
+/// The existing tolerance of a non-distinct `sum`, `min` or `max` predates this
+/// and is left alone: narrowing it would change plans that have always been
+/// rewritten, which no measurement here calls for.
+fn rewrite_pays_for_count(
+ distinct_aggs: &[(&Arc<AggregateUDF>, &Vec<Expr>)],
Review Comment:
Small cleanup: I don't think this needs to require `Vec<Expr>`. Could this take slices instead, for example `&[(&Arc<AggregateUDF>, &[Expr])]`, and use `args.as_slice()` when collecting? That seems a little more idiomatic and matches the surrounding slice-based APIs. No behavior change intended.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#discussion_r3958017269
##########
datafusion/optimizer/src/single_distinct_to_groupby.rs:
##########
@@ -96,7 +170,58 @@ fn is_single_distinct_agg(aggr_expr: &[Expr]) -> Result<bool> {
return Ok(false);
}
}
- Ok(aggregate_count == aggr_expr.len() && fields_set.len() == 1)
+ if aggregate_count != aggr_expr.len() || fields_set.len() != 1 {
+ return Ok(false);
+ }
+ if has_count_rollup && !rewrite_pays_for_count(&distinct_aggs, input_schema)? {
+ return Ok(false);
+ }
+ Ok(true)
+}
+
+/// Whether the rewrite is worth extending to a plan that only qualifies because
+/// of the non-distinct `count`.
+///
+/// The rewrite is not free: every other aggregate moves down to the inner group
+/// by, which has a row per `(group, distinct value)` pair rather than per group,
+/// and each one keeps its state at that finer grain. What pays for it is taking
+/// the distinct aggregate off `GroupsAccumulatorAdapter`, whose one boxed
+/// accumulator per group is the expensive shape. A distinct aggregate that
+/// already has a specialized `GroupsAccumulator` never went near the adapter, so
+/// there is nothing to buy and only the inner group by to pay for: ClickBench
+/// Q22, whose `count(DISTINCT "UserID")` is over an `Int64`, measured a 132%
+/// increase in peak memory pool reservation when the rewrite applied to it.
+///
+/// `Some(false)` is the only answer that buys anything. `Some(true)` says the
+/// call never reaches the adapter. `None` says the function does not answer the
+/// question from the argument types, which is the default and so the answer for
+/// almost every function. Silence is not evidence, and reading it as `Some(false)`
+/// would open this path to every such function: `sum(DISTINCT int_col)` beside a
+/// `count(*)` measured 3.15x the peak memory once rewritten, over 4,000,000 rows
+/// in 2,000 groups.
Review Comment:
Instead of citing numbers here which can quickly become outdated we should link to the PR where this was measured as a point in time artifact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5514794707
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5514755290-2105-kkpnq 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5514789313
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5514755290-2103-9pm4r 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5500259394
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5500225568)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5500225568-2072-x9l4r 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (f47c04523f7f84ad0993001b47ae1bc2ce92b0d5) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..f47c04523f7f84ad0993001b47ae1bc2ce92b0d5)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290
run benchmark clickbench_partitioned clickbench_partitioned clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #24859:
URL: https://github.com/apache/datafusion/pull/24859
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5574029104
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 757.16 ms β 781.13 ms β no change β
β QQuery 1 β 188.91 ms β 190.00 ms β no change β
β QQuery 2 β 453.34 ms β 451.41 ms β no change β
β QQuery 3 β 313.91 ms β 316.10 ms β no change β
β QQuery 4 β 1128.57 ms β 1141.32 ms β no change β
β QQuery 5 β 17994.72 ms β 18120.21 ms β no change β
β QQuery 6 β 2.69 ms β 2.61 ms β no change β
β QQuery 7 β 662.47 ms β 668.71 ms β no change β
β QQuery 8 β 416.00 ms β 413.32 ms β no change β
β QQuery 9 β 2672.71 ms β 2732.93 ms β no change β
β QQuery 10 β 635.32 ms β 634.05 ms β no change β
β QQuery 11 β 2330.55 ms β 2243.59 ms β no change β
β QQuery 12 β 187.47 ms β 190.77 ms β no change β
β QQuery 13 β 548.08 ms β 552.43 ms β no change β
β QQuery 14 β 2471.40 ms β 546.00 ms β +4.53x faster β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 30763.30ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 28984.58ms β
β Average Time (HEAD) β 2050.89ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1932.31ms β
β Queries Faster β 1 β
β Queries Slower β 0 β
β Queries with No Change β 14 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 757.16 / 770.50 Β±10.17 / 785.26 ms β 781.13 / 795.13 Β±9.29 / 809.50 ms β no change β
β QQuery 1 β 188.91 / 191.41 Β±3.20 / 197.75 ms β 190.00 / 190.52 Β±0.46 / 191.08 ms β no change β
β QQuery 2 β 453.34 / 456.62 Β±2.60 / 459.57 ms β 451.41 / 454.89 Β±2.02 / 457.39 ms β no change β
β QQuery 3 β 313.91 / 315.47 Β±0.97 / 316.75 ms β 316.10 / 317.76 Β±1.08 / 319.24 ms β no change β
β QQuery 4 β 1128.57 / 1150.51 Β±15.01 / 1174.14 ms β 1141.32 / 1151.15 Β±10.63 / 1165.50 ms β no change β
β QQuery 5 β 17994.72 / 18400.60 Β±253.34 / 18711.10 ms β 18120.21 / 18754.14 Β±510.55 / 19288.80 ms β no change β
β QQuery 6 β 2.69 / 2.90 Β±0.29 / 3.47 ms β 2.61 / 2.84 Β±0.37 / 3.57 ms β no change β
β QQuery 7 β 662.47 / 744.05 Β±53.47 / 830.47 ms β 668.71 / 725.76 Β±31.09 / 754.32 ms β no change β
β QQuery 8 β 416.00 / 432.46 Β±22.68 / 477.28 ms β 413.32 / 445.81 Β±44.75 / 530.08 ms β no change β
β QQuery 9 β 2672.71 / 2853.13 Β±108.13 / 2960.09 ms β 2732.93 / 2856.65 Β±63.01 / 2902.26 ms β no change β
β QQuery 10 β 635.32 / 670.79 Β±20.75 / 697.31 ms β 634.05 / 667.64 Β±44.08 / 753.85 ms β no change β
β QQuery 11 β 2330.55 / 2381.20 Β±37.51 / 2425.01 ms β 2243.59 / 2285.15 Β±22.83 / 2311.16 ms β no change β
β QQuery 12 β 187.47 / 229.90 Β±75.39 / 380.52 ms β 190.77 / 218.20 Β±43.41 / 304.55 ms β +1.05x faster β
β QQuery 13 β 548.08 / 569.99 Β±16.41 / 598.93 ms β 552.43 / 558.91 Β±4.97 / 567.12 ms β no change β
β QQuery 14 β 2471.40 / 2529.67 Β±38.65 / 2583.56 ms β 546.00 / 624.86 Β±65.79 / 717.10 ms β +4.05x faster β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 31699.20ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 30049.42ms β
β Average Time (HEAD) β 2113.28ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 2003.29ms β
β Queries Faster β 2 β
β Queries Slower β 0 β
β Queries with No Change β 13 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `16ace4f (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_extended`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 844.5 MiB | 863.0 MiB | +2.2% |
| Query 1 | 2.4 MiB | 2.4 MiB | +0.0% |
| Query 2 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 3 | 11.7 MiB | 11.7 MiB | +0.0% |
| Query 4 | 4.2 GiB | 4.2 GiB | +0.0% |
| Query 5 | 5.5 GiB | 5.5 GiB | -0.1% |
| Query 6 | 104 B | 104 B | +0.0% |
| Query 7 | 4.9 GiB | 4.9 GiB | -0.1% |
| Query 8 | 37.0 MiB | 37.0 MiB | +0.0% |
| Query 9 | 4.8 GiB | 4.7 GiB | -1.8% |
| Query 10 | 2.3 MiB | 1.9 MiB | -17.9% |
| Query 11 | 2.8 GiB | 2.8 GiB | -0.2% |
| Query 12 | 1.1 MiB | 1.4 MiB | +23.8% |
| Query 13 | 520 B | 520 B | +0.0% |
| Query 14 | 4.4 GiB | 1.3 GiB | -70.5% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_extended | base (`16ace4f (merge-base)`) | 5.5 GiB | 23.1 GiB | 17.6 GiB | 4.2Γ |
| clickbench_extended | changed (`claude/single-distinct-to-groupby-allow-count`) | 5.5 GiB | 24.3 GiB | 18.8 GiB | 4.4Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 160.0s |
| Peak memory | 23.1 GiB |
| Avg memory | 5.5 GiB |
| CPU user | 1510.3s |
| CPU sys | 90.3s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 155.0s |
| Peak memory | 24.3 GiB |
| Avg memory | 5.3 GiB |
| CPU user | 1441.7s |
| CPU sys | 86.2s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5566643664
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5566611080-2196-nd76m 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5567063164
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 813.01 ms β 777.37 ms β no change β
β QQuery 1 β 188.04 ms β 188.25 ms β no change β
β QQuery 2 β 449.27 ms β 449.91 ms β no change β
β QQuery 3 β 314.49 ms β 316.07 ms β no change β
β QQuery 4 β 1142.10 ms β 1148.43 ms β no change β
β QQuery 5 β 10986.58 ms β 10600.59 ms β no change β
β QQuery 6 β 2.63 ms β 2.76 ms β no change β
β QQuery 7 β 693.61 ms β 692.21 ms β no change β
β QQuery 8 β 410.71 ms β 423.30 ms β no change β
β QQuery 9 β 2948.99 ms β 3231.74 ms β 1.10x slower β
β QQuery 10 β 623.52 ms β 643.95 ms β no change β
β QQuery 11 β 2234.22 ms β 2120.47 ms β +1.05x faster β
β QQuery 12 β 186.08 ms β 194.65 ms β no change β
β QQuery 13 β 535.40 ms β 543.05 ms β no change β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 21528.64ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 21332.74ms β
β Average Time (HEAD) β 1537.76ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1523.77ms β
β Queries Faster β 1 β
β Queries Slower β 1 β
β Queries with No Change β 12 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 813.01 / 830.45 Β±10.39 / 841.65 ms β 777.37 / 797.48 Β±14.15 / 816.71 ms β no change β
β QQuery 1 β 188.04 / 190.24 Β±3.54 / 197.29 ms β 188.25 / 192.24 Β±6.75 / 205.73 ms β no change β
β QQuery 2 β 449.27 / 457.13 Β±5.59 / 464.90 ms β 449.91 / 451.88 Β±1.56 / 454.33 ms β no change β
β QQuery 3 β 314.49 / 316.01 Β±0.86 / 316.87 ms β 316.07 / 317.54 Β±0.95 / 318.81 ms β no change β
β QQuery 4 β 1142.10 / 1181.35 Β±27.19 / 1226.11 ms β 1148.43 / 1161.71 Β±7.92 / 1172.70 ms β no change β
β QQuery 5 β 10986.58 / 11188.78 Β±179.01 / 11421.09 ms β 10600.59 / 11157.86 Β±458.18 / 11827.43 ms β no change β
β QQuery 6 β 2.63 / 2.98 Β±0.35 / 3.64 ms β 2.76 / 2.99 Β±0.32 / 3.58 ms β no change β
β QQuery 7 β 693.61 / 755.50 Β±65.69 / 869.85 ms β 692.21 / 726.53 Β±22.67 / 762.57 ms β no change β
β QQuery 8 β 410.71 / 420.67 Β±6.27 / 427.32 ms β 423.30 / 446.36 Β±27.48 / 496.79 ms β 1.06x slower β
β QQuery 9 β 2948.99 / 3003.73 Β±36.37 / 3054.82 ms β 3231.74 / 3280.54 Β±40.86 / 3348.53 ms β 1.09x slower β
β QQuery 10 β 623.52 / 701.15 Β±77.12 / 843.95 ms β 643.95 / 713.73 Β±101.38 / 915.04 ms β no change β
β QQuery 11 β 2234.22 / 2350.58 Β±63.81 / 2414.74 ms β 2120.47 / 2253.85 Β±100.25 / 2359.84 ms β no change β
β QQuery 12 β 186.08 / 223.71 Β±68.71 / 361.01 ms β 194.65 / 219.95 Β±31.15 / 279.52 ms β no change β
β QQuery 13 β 535.40 / 550.32 Β±9.48 / 563.50 ms β 543.05 / 569.95 Β±25.23 / 606.73 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 22172.62ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 22292.60ms β
β Average Time (HEAD) β 1583.76ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1592.33ms β
β Queries Faster β 0 β
β Queries Slower β 2 β
β Queries with No Change β 12 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 12.2 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 1033.7s |
| CPU sys | 47.2s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 11.8 GiB |
| Avg memory | 5.0 GiB |
| CPU user | 1036.0s |
| CPU sys | 50.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5500500295
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5500225568)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (f47c04523f7f84ad0993001b47ae1bc2ce92b0d5) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..f47c04523f7f84ad0993001b47ae1bc2ce92b0d5)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.38 ms β 1.22 ms β +1.12x faster β
β QQuery 1 β 12.34 ms β 11.86 ms β no change β
β QQuery 2 β 37.35 ms β 36.73 ms β no change β
β QQuery 3 β 33.25 ms β 31.14 ms β +1.07x faster β
β QQuery 4 β 265.05 ms β 221.44 ms β +1.20x faster β
β QQuery 5 β 276.49 ms β 269.41 ms β no change β
β QQuery 6 β 1.28 ms β 1.28 ms β no change β
β QQuery 7 β 13.28 ms β 13.09 ms β no change β
β QQuery 8 β 337.55 ms β 330.68 ms β no change β
β QQuery 9 β 452.80 ms β 447.81 ms β no change β
β QQuery 10 β 69.67 ms β 69.38 ms β no change β
β QQuery 11 β 80.87 ms β 80.40 ms β no change β
β QQuery 12 β 265.72 ms β 266.16 ms β no change β
β QQuery 13 β 978.95 ms β 959.23 ms β no change β
β QQuery 14 β 281.40 ms β 287.72 ms β no change β
β QQuery 15 β 266.83 ms β 260.92 ms β no change β
β QQuery 16 β 1228.08 ms β 1196.31 ms β no change β
β QQuery 17 β 916.26 ms β 890.67 ms β no change β
β QQuery 18 β 2497.04 ms β 2464.68 ms β no change β
β QQuery 19 β 28.04 ms β 30.09 ms β 1.07x slower β
β QQuery 20 β 528.22 ms β 524.86 ms β no change β
β QQuery 21 β 516.39 ms β 512.66 ms β no change β
β QQuery 22 β 980.18 ms β 974.68 ms β no change β
β QQuery 23 β 3061.15 ms β 3010.47 ms β no change β
β QQuery 24 β 42.00 ms β 41.44 ms β no change β
β QQuery 25 β 110.73 ms β 109.66 ms β no change β
β QQuery 26 β 42.25 ms β 41.12 ms β no change β
β QQuery 27 β 511.60 ms β 509.48 ms β no change β
β QQuery 28 β 2913.69 ms β 2885.57 ms β no change β
β QQuery 29 β 41.01 ms β 41.29 ms β no change β
β QQuery 30 β 298.78 ms β 296.46 ms β no change β
β QQuery 31 β 273.81 ms β 284.70 ms β no change β
β QQuery 32 β 3254.62 ms β 3309.85 ms β no change β
β QQuery 33 β 2515.41 ms β 2534.95 ms β no change β
β QQuery 34 β 2659.40 ms β 2562.43 ms β no change β
β QQuery 35 β 280.03 ms β 278.34 ms β no change β
β QQuery 36 β 65.80 ms β 65.81 ms β no change β
β QQuery 37 β 35.17 ms β 35.47 ms β no change β
β QQuery 38 β 39.97 ms β 40.54 ms β no change β
β QQuery 39 β 133.36 ms β 130.60 ms β no change β
β QQuery 40 β 13.78 ms β 13.90 ms β no change β
β QQuery 41 β 13.60 ms β 13.63 ms β no change β
β QQuery 42 β 12.82 ms β 13.08 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26387.39ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 26101.22ms β
β Average Time (HEAD) β 613.66ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 607.01ms β
β Queries Faster β 3 β
β Queries Slower β 1 β
β Queries with No Change β 39 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.38 / 4.40 Β±5.86 / 16.12 ms β 1.22 / 3.94 Β±5.36 / 14.66 ms β +1.12x faster β
β QQuery 1 β 12.34 / 12.80 Β±0.27 / 13.14 ms β 11.86 / 12.01 Β±0.12 / 12.20 ms β +1.07x faster β
β QQuery 2 β 37.35 / 37.99 Β±0.55 / 38.93 ms β 36.73 / 36.90 Β±0.12 / 37.05 ms β no change β
β QQuery 3 β 33.25 / 34.03 Β±0.82 / 35.47 ms β 31.14 / 31.35 Β±0.14 / 31.54 ms β +1.09x faster β
β QQuery 4 β 265.05 / 269.97 Β±4.66 / 276.23 ms β 221.44 / 231.96 Β±8.16 / 245.31 ms β +1.16x faster β
β QQuery 5 β 276.49 / 288.95 Β±13.08 / 309.95 ms β 269.41 / 274.36 Β±4.16 / 280.38 ms β +1.05x faster β
β QQuery 6 β 1.28 / 1.41 Β±0.20 / 1.81 ms β 1.28 / 1.44 Β±0.24 / 1.91 ms β no change β
β QQuery 7 β 13.28 / 13.35 Β±0.06 / 13.47 ms β 13.09 / 13.20 Β±0.07 / 13.29 ms β no change β
β QQuery 8 β 337.55 / 351.39 Β±10.86 / 361.65 ms β 330.68 / 414.89 Β±159.42 / 733.70 ms β 1.18x slower β
β QQuery 9 β 452.80 / 476.14 Β±19.03 / 503.53 ms β 447.81 / 454.59 Β±6.52 / 466.88 ms β no change β
β QQuery 10 β 69.67 / 75.80 Β±7.42 / 87.62 ms β 69.38 / 72.74 Β±4.97 / 82.59 ms β no change β
β QQuery 11 β 80.87 / 81.53 Β±0.43 / 82.17 ms β 80.40 / 81.73 Β±2.17 / 86.04 ms β no change β
β QQuery 12 β 265.72 / 273.17 Β±6.19 / 281.39 ms β 266.16 / 269.76 Β±3.69 / 275.76 ms β no change β
β QQuery 13 β 978.95 / 984.79 Β±4.26 / 992.10 ms β 959.23 / 972.59 Β±11.09 / 989.55 ms β no change β
β QQuery 14 β 281.40 / 289.20 Β±5.44 / 297.13 ms β 287.72 / 311.18 Β±15.57 / 333.71 ms β 1.08x slower β
β QQuery 15 β 266.83 / 271.93 Β±2.95 / 275.50 ms β 260.92 / 274.45 Β±10.73 / 292.43 ms β no change β
β QQuery 16 β 1228.08 / 1270.69 Β±27.81 / 1314.41 ms β 1196.31 / 1231.12 Β±19.58 / 1251.73 ms β no change β
β QQuery 17 β 916.26 / 944.27 Β±23.43 / 979.60 ms β 890.67 / 931.33 Β±22.36 / 957.69 ms β no change β
β QQuery 18 β 2497.04 / 2549.71 Β±59.07 / 2659.37 ms β 2464.68 / 2619.87 Β±114.28 / 2736.65 ms β no change β
β QQuery 19 β 28.04 / 29.82 Β±2.46 / 34.68 ms β 30.09 / 30.96 Β±0.56 / 31.67 ms β no change β
β QQuery 20 β 528.22 / 535.30 Β±4.58 / 540.87 ms β 524.86 / 548.63 Β±18.95 / 574.17 ms β no change β
β QQuery 21 β 516.39 / 520.17 Β±3.16 / 525.39 ms β 512.66 / 517.24 Β±4.36 / 525.02 ms β no change β
β QQuery 22 β 980.18 / 986.55 Β±4.83 / 992.72 ms β 974.68 / 988.63 Β±9.98 / 1003.70 ms β no change β
β QQuery 23 β 3061.15 / 3176.07 Β±88.50 / 3302.98 ms β 3010.47 / 3045.11 Β±30.25 / 3090.12 ms β no change β
β QQuery 24 β 42.00 / 42.34 Β±0.44 / 43.20 ms β 41.44 / 43.22 Β±3.08 / 49.38 ms β no change β
β QQuery 25 β 110.73 / 116.59 Β±7.38 / 130.21 ms β 109.66 / 113.03 Β±3.80 / 120.17 ms β no change β
β QQuery 26 β 42.25 / 43.27 Β±0.79 / 44.67 ms β 41.12 / 41.64 Β±0.66 / 42.93 ms β no change β
β QQuery 27 β 511.60 / 515.42 Β±3.73 / 522.37 ms β 509.48 / 514.21 Β±4.78 / 521.97 ms β no change β
β QQuery 28 β 2913.69 / 2942.96 Β±24.29 / 2976.11 ms β 2885.57 / 2943.07 Β±47.48 / 3016.49 ms β no change β
β QQuery 29 β 41.01 / 52.70 Β±10.75 / 69.66 ms β 41.29 / 46.47 Β±8.04 / 62.43 ms β +1.13x faster β
β QQuery 30 β 298.78 / 305.77 Β±4.66 / 313.00 ms β 296.46 / 317.23 Β±30.89 / 378.34 ms β no change β
β QQuery 31 β 273.81 / 287.28 Β±7.37 / 294.32 ms β 284.70 / 291.53 Β±4.80 / 296.70 ms β no change β
β QQuery 32 β 3254.62 / 3304.71 Β±50.88 / 3397.00 ms β 3309.85 / 3514.92 Β±160.53 / 3712.98 ms β 1.06x slower β
β QQuery 33 β 2515.41 / 2697.29 Β±162.83 / 2974.36 ms β 2534.95 / 2607.56 Β±78.80 / 2758.34 ms β no change β
β QQuery 34 β 2659.40 / 2764.19 Β±119.47 / 2950.04 ms β 2562.43 / 2636.57 Β±48.78 / 2684.60 ms β no change β
β QQuery 35 β 280.03 / 288.16 Β±6.99 / 297.41 ms β 278.34 / 289.88 Β±10.71 / 308.26 ms β no change β
β QQuery 36 β 65.80 / 72.10 Β±3.65 / 75.62 ms β 65.81 / 69.11 Β±2.82 / 72.45 ms β no change β
β QQuery 37 β 35.17 / 35.91 Β±0.46 / 36.61 ms β 35.47 / 44.91 Β±17.07 / 79.01 ms β 1.25x slower β
β QQuery 38 β 39.97 / 54.20 Β±24.14 / 102.34 ms β 40.54 / 43.23 Β±1.55 / 45.17 ms β +1.25x faster β
β QQuery 39 β 133.36 / 140.02 Β±4.66 / 147.49 ms β 130.60 / 134.20 Β±3.25 / 140.32 ms β no change β
β QQuery 40 β 13.78 / 14.17 Β±0.25 / 14.52 ms β 13.90 / 14.46 Β±0.38 / 15.06 ms β no change β
β QQuery 41 β 13.60 / 13.82 Β±0.18 / 14.12 ms β 13.63 / 14.37 Β±0.88 / 16.03 ms β no change β
β QQuery 42 β 12.82 / 18.13 Β±9.98 / 38.07 ms β 13.08 / 14.22 Β±1.63 / 17.42 ms β +1.27x faster β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 27188.49ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 27063.85ms β
β Average Time (HEAD) β 632.29ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 629.39ms β
β Queries Faster β 8 β
β Queries Slower β 4 β
β Queries with No Change β 31 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `da89c7c (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 757.7 MiB | 741.3 MiB | -2.2% |
| Query 5 | 1.2 GiB | 1.2 GiB | -0.4% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 40.1 MiB | 60.2 MiB | +50.1% |
| Query 8 | 873.1 MiB | 852.4 MiB | -2.4% |
| Query 9 | 551.7 MiB | 551.7 MiB | +0.0% |
| Query 10 | 107.6 MiB | 111.0 MiB | +3.2% |
| Query 11 | 113.2 MiB | 115.9 MiB | +2.4% |
| Query 12 | 1.3 GiB | 1.3 GiB | -0.2% |
| Query 13 | 998.5 MiB | 1.1 GiB | +8.5% |
| Query 14 | 1.3 GiB | 1.3 GiB | -1.3% |
| Query 15 | 1.1 GiB | 1.2 GiB | +1.9% |
| Query 16 | 1.9 GiB | 1.7 GiB | -10.3% |
| Query 17 | 2.0 GiB | 1.9 GiB | -1.2% |
| Query 18 | 2.0 GiB | 2.0 GiB | -2.9% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.3 MiB | -0.0% |
| Query 22 | 3.1 MiB | 7.3 MiB | +132.2% |
| Query 23 | 26.8 MiB | 25.7 MiB | -4.0% |
| Query 24 | 58.8 MiB | 58.6 MiB | -0.4% |
| Query 25 | 170.3 MiB | 173.2 MiB | +1.7% |
| Query 26 | 59.9 MiB | 60.3 MiB | +0.6% |
| Query 27 | 2.2 MiB | 2.2 MiB | +0.0% |
| Query 28 | 1.5 GiB | 1.4 GiB | -3.7% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 698.5 MiB | 707.1 MiB | +1.2% |
| Query 31 | 1.5 GiB | 1.5 GiB | +3.2% |
| Query 32 | 926.9 MiB | 966.9 MiB | +4.3% |
| Query 33 | 2.0 GiB | 2.1 GiB | +5.2% |
| Query 34 | 2.1 GiB | 2.1 GiB | -2.6% |
| Query 35 | 612.0 MiB | 622.3 MiB | +1.7% |
| Query 36 | 113.6 MiB | 111.1 MiB | -2.2% |
| Query 37 | 6.9 MiB | 6.9 MiB | +0.0% |
| Query 38 | 5.2 MiB | 5.2 MiB | -0.4% |
| Query 39 | 297.5 MiB | 297.8 MiB | +0.1% |
| Query 40 | 2.0 MiB | 1.8 MiB | -8.3% |
| Query 41 | 3.1 MiB | 3.1 MiB | +0.0% |
| Query 42 | 1.6 MiB | 1.6 MiB | +5.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`da89c7c (merge-base)`) | 2.1 GiB | 8.7 GiB | 6.6 GiB | 4.1Γ |
| clickbench_partitioned | changed (`claude/single-distinct-to-groupby-allow-count`) | 2.1 GiB | 8.6 GiB | 6.4 GiB | 4.0Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 8.7 GiB |
| Avg memory | 5.3 GiB |
| CPU user | 1382.5s |
| CPU sys | 131.6s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 8.6 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1376.6s |
| CPU sys | 132.7s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080
run benchmark clickbench_extended clickbench_extended clickbench_extended
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5585614783
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573827429
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5573802227-2205-jcl4f 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573809747
Hi @adriangb, you asked to view the benchmark queue (https://github.com/apache/datafusion/pull/24859#issuecomment-5573809398).
| Comment | Repo | PR | User | Benchmarks | Status |
| --- | --- | --- | --- | --- | --- |
| [#5573802227](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227) | apache/datafusion | #24859 | adriangb | ["clickbench_extended"] | running |
| [#5573802227](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227) | apache/datafusion | #24859 | adriangb | ["clickbench_extended"] | running |
| [#5573802227](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227) | apache/datafusion | #24859 | adriangb | ["clickbench_extended"] | running |
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227
run benchmark clickbench_extended clickbench_extended clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 16G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573825699
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5573802227)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5573802227-2207-k5kvb 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (07f24b63e9475c139f2d2ff8fc7b270c3cd17955) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..07f24b63e9475c139f2d2ff8fc7b270c3cd17955)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "16G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5573809398
show benchmark queue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5567059817
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5566611080)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (3d3cf6b581eafdb47d91c63511d39c83fddf6d91) to e1ca94f (merge-base) [diff](https://github.com/apache/datafusion/compare/e1ca94fb11ab0f2ead9bddd2b5be6666e5d75c62..3d3cf6b581eafdb47d91c63511d39c83fddf6d91)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_extended
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 780.26 ms β 783.34 ms β no change β
β QQuery 1 β 188.77 ms β 188.75 ms β no change β
β QQuery 2 β 449.28 ms β 447.96 ms β no change β
β QQuery 3 β 314.27 ms β 313.49 ms β no change β
β QQuery 4 β 1149.65 ms β 1159.99 ms β no change β
β QQuery 5 β 11158.56 ms β 10462.77 ms β +1.07x faster β
β QQuery 6 β 2.66 ms β 2.64 ms β no change β
β QQuery 7 β 689.69 ms β 690.22 ms β no change β
β QQuery 8 β 407.32 ms β 415.12 ms β no change β
β QQuery 9 β 2882.30 ms β 3194.95 ms β 1.11x slower β
β QQuery 10 β 634.94 ms β 644.54 ms β no change β
β QQuery 11 β 2260.97 ms β 2127.60 ms β +1.06x faster β
β QQuery 12 β 189.48 ms β 188.30 ms β no change β
β QQuery 13 β 545.43 ms β 549.45 ms β no change β
βββββββββββββ΄ββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 21653.60ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 21169.11ms β
β Average Time (HEAD) β 1546.69ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1512.08ms β
β Queries Faster β 2 β
β Queries Slower β 1 β
β Queries with No Change β 11 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_extended.json
--------------------
βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 780.26 / 788.40 Β±7.00 / 798.02 ms β 783.34 / 796.55 Β±10.67 / 809.75 ms β no change β
β QQuery 1 β 188.77 / 190.78 Β±3.82 / 198.42 ms β 188.75 / 194.28 Β±5.60 / 204.62 ms β no change β
β QQuery 2 β 449.28 / 453.47 Β±2.78 / 457.12 ms β 447.96 / 452.87 Β±2.56 / 455.13 ms β no change β
β QQuery 3 β 314.27 / 315.33 Β±0.55 / 315.88 ms β 313.49 / 315.12 Β±1.11 / 316.35 ms β no change β
β QQuery 4 β 1149.65 / 1161.28 Β±8.39 / 1169.85 ms β 1159.99 / 1169.81 Β±9.52 / 1186.09 ms β no change β
β QQuery 5 β 11158.56 / 11546.89 Β±385.86 / 12210.52 ms β 10462.77 / 11461.59 Β±673.70 / 12207.74 ms β no change β
β QQuery 6 β 2.66 / 2.90 Β±0.30 / 3.49 ms β 2.64 / 2.85 Β±0.31 / 3.47 ms β no change β
β QQuery 7 β 689.69 / 757.69 Β±57.76 / 848.42 ms β 690.22 / 740.91 Β±43.04 / 806.86 ms β no change β
β QQuery 8 β 407.32 / 439.56 Β±27.83 / 472.94 ms β 415.12 / 432.21 Β±33.30 / 498.81 ms β no change β
β QQuery 9 β 2882.30 / 2984.86 Β±74.40 / 3111.76 ms β 3194.95 / 3221.07 Β±20.22 / 3251.69 ms β 1.08x slower β
β QQuery 10 β 634.94 / 688.29 Β±74.43 / 830.48 ms β 644.54 / 662.63 Β±20.03 / 700.78 ms β no change β
β QQuery 11 β 2260.97 / 2394.00 Β±67.19 / 2439.08 ms β 2127.60 / 2229.93 Β±71.74 / 2292.79 ms β +1.07x faster β
β QQuery 12 β 189.48 / 211.00 Β±17.54 / 242.62 ms β 188.30 / 238.67 Β±92.46 / 423.49 ms β 1.13x slower β
β QQuery 13 β 545.43 / 555.20 Β±7.18 / 564.94 ms β 549.45 / 567.57 Β±16.61 / 590.64 ms β no change β
βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 22489.65ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 22486.06ms β
β Average Time (HEAD) β 1606.40ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 1606.15ms β
β Queries Faster β 1 β
β Queries Slower β 2 β
β Queries with No Change β 11 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_extended β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 11.8 GiB |
| Avg memory | 5.2 GiB |
| CPU user | 1033.4s |
| CPU sys | 49.0s |
| Peak spill | 0 B |
**clickbench_extended β branch**
| Metric | Value |
|--------|-------|
| Wall time | 115.0s |
| Peak memory | 11.6 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1030.0s |
| CPU sys | 50.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5585656009
π€ Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5585614783)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5585614783-2212-wvrbr 6.12.94+ #1 SMP Fri Jul 17 09:42:57 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing claude/single-distinct-to-groupby-allow-count (173ae06366a11599f7f443382f7b27ca57854df2) to 16ace4f (merge-base) [diff](https://github.com/apache/datafusion/compare/16ace4f3e5f41cde80a5249fbeb6330d222fd8b1..173ae06366a11599f7f443382f7b27ca57854df2)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5586634401
Okay @kosiew this is ready now. Could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5500225568
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 4G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5515049260
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.27 ms β 1.22 ms β no change β
β QQuery 1 β 12.22 ms β 11.62 ms β no change β
β QQuery 2 β 38.04 ms β 36.38 ms β no change β
β QQuery 3 β 32.62 ms β 30.96 ms β +1.05x faster β
β QQuery 4 β 227.33 ms β 220.88 ms β no change β
β QQuery 5 β 274.86 ms β 270.90 ms β no change β
β QQuery 6 β 1.28 ms β 1.25 ms β no change β
β QQuery 7 β 13.55 ms β 13.08 ms β no change β
β QQuery 8 β 332.26 ms β 327.53 ms β no change β
β QQuery 9 β 446.62 ms β 457.33 ms β no change β
β QQuery 10 β 69.37 ms β 69.51 ms β no change β
β QQuery 11 β 82.09 ms β 80.21 ms β no change β
β QQuery 12 β 269.35 ms β 266.19 ms β no change β
β QQuery 13 β 966.91 ms β 966.01 ms β no change β
β QQuery 14 β 284.98 ms β 319.46 ms β 1.12x slower β
β QQuery 15 β 276.84 ms β 312.31 ms β 1.13x slower β
β QQuery 16 β 1238.76 ms β 1256.69 ms β no change β
β QQuery 17 β 898.65 ms β 940.45 ms β no change β
β QQuery 18 β 2455.27 ms β 2464.09 ms β no change β
β QQuery 19 β 29.77 ms β 27.66 ms β +1.08x faster β
β QQuery 20 β 517.52 ms β 510.92 ms β no change β
β QQuery 21 β 518.26 ms β 509.53 ms β no change β
β QQuery 22 β 977.48 ms β 977.16 ms β no change β
β QQuery 23 β 3000.63 ms β 2958.54 ms β no change β
β QQuery 24 β 40.79 ms β 41.28 ms β no change β
β QQuery 25 β 109.83 ms β 108.64 ms β no change β
β QQuery 26 β 41.31 ms β 41.13 ms β no change β
β QQuery 27 β 508.08 ms β 508.58 ms β no change β
β QQuery 28 β 2907.16 ms β 2980.89 ms β no change β
β QQuery 29 β 41.41 ms β 41.08 ms β no change β
β QQuery 30 β 302.75 ms β 297.41 ms β no change β
β QQuery 31 β 273.33 ms β 280.18 ms β no change β
β QQuery 32 β 3282.50 ms β 3248.08 ms β no change β
β QQuery 33 β 2590.55 ms β 2520.60 ms β no change β
β QQuery 34 β 2692.59 ms β 2580.73 ms β no change β
β QQuery 35 β 277.90 ms β 300.20 ms β 1.08x slower β
β QQuery 36 β 67.22 ms β 70.81 ms β 1.05x slower β
β QQuery 37 β 35.86 ms β 38.17 ms β 1.06x slower β
β QQuery 38 β 40.22 ms β 42.64 ms β 1.06x slower β
β QQuery 39 β 130.00 ms β 159.50 ms β 1.23x slower β
β QQuery 40 β 13.96 ms β 16.06 ms β 1.15x slower β
β QQuery 41 β 13.59 ms β 15.23 ms β 1.12x slower β
β QQuery 42 β 13.39 ms β 14.62 ms β 1.09x slower β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26348.35ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 26335.75ms β
β Average Time (HEAD) β 612.75ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 612.46ms β
β Queries Faster β 2 β
β Queries Slower β 10 β
β Queries with No Change β 31 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.27 / 4.22 Β±5.73 / 15.68 ms β 1.22 / 3.95 Β±5.39 / 14.73 ms β +1.07x faster β
β QQuery 1 β 12.22 / 12.58 Β±0.25 / 12.87 ms β 11.62 / 12.05 Β±0.27 / 12.45 ms β no change β
β QQuery 2 β 38.04 / 38.46 Β±0.31 / 38.92 ms β 36.38 / 36.55 Β±0.24 / 37.01 ms β no change β
β QQuery 3 β 32.62 / 33.85 Β±1.02 / 35.31 ms β 30.96 / 31.38 Β±0.61 / 32.58 ms β +1.08x faster β
β QQuery 4 β 227.33 / 229.65 Β±1.63 / 232.07 ms β 220.88 / 225.64 Β±3.37 / 231.19 ms β no change β
β QQuery 5 β 274.86 / 277.89 Β±1.70 / 279.65 ms β 270.90 / 274.70 Β±3.88 / 281.59 ms β no change β
β QQuery 6 β 1.28 / 1.46 Β±0.21 / 1.85 ms β 1.25 / 1.39 Β±0.22 / 1.82 ms β no change β
β QQuery 7 β 13.55 / 13.67 Β±0.10 / 13.85 ms β 13.08 / 13.21 Β±0.09 / 13.36 ms β no change β
β QQuery 8 β 332.26 / 340.41 Β±6.04 / 347.47 ms β 327.53 / 335.83 Β±5.09 / 341.12 ms β no change β
β QQuery 9 β 446.62 / 459.03 Β±7.80 / 466.14 ms β 457.33 / 467.33 Β±11.39 / 488.93 ms β no change β
β QQuery 10 β 69.37 / 70.42 Β±0.88 / 72.00 ms β 69.51 / 70.92 Β±1.98 / 74.83 ms β no change β
β QQuery 11 β 82.09 / 82.50 Β±0.30 / 82.94 ms β 80.21 / 81.06 Β±0.87 / 82.12 ms β no change β
β QQuery 12 β 269.35 / 271.44 Β±2.11 / 275.24 ms β 266.19 / 272.74 Β±3.97 / 278.22 ms β no change β
β QQuery 13 β 966.91 / 975.94 Β±6.10 / 982.32 ms β 966.01 / 986.04 Β±21.18 / 1026.61 ms β no change β
β QQuery 14 β 284.98 / 314.15 Β±16.60 / 330.66 ms β 319.46 / 328.91 Β±8.76 / 344.68 ms β no change β
β QQuery 15 β 276.84 / 298.01 Β±18.55 / 320.53 ms β 312.31 / 317.90 Β±5.32 / 327.14 ms β 1.07x slower β
β QQuery 16 β 1238.76 / 1273.35 Β±33.77 / 1337.52 ms β 1256.69 / 1342.81 Β±52.12 / 1416.22 ms β 1.05x slower β
β QQuery 17 β 898.65 / 921.54 Β±16.68 / 941.53 ms β 940.45 / 998.66 Β±39.90 / 1060.94 ms β 1.08x slower β
β QQuery 18 β 2455.27 / 2599.91 Β±116.87 / 2750.73 ms β 2464.09 / 2541.95 Β±69.68 / 2672.30 ms β no change β
β QQuery 19 β 29.77 / 30.61 Β±0.77 / 31.58 ms β 27.66 / 28.22 Β±0.55 / 29.20 ms β +1.08x faster β
β QQuery 20 β 517.52 / 523.78 Β±4.51 / 531.10 ms β 510.92 / 524.84 Β±8.84 / 534.47 ms β no change β
β QQuery 21 β 518.26 / 530.80 Β±8.21 / 540.00 ms β 509.53 / 525.11 Β±11.95 / 545.07 ms β no change β
β QQuery 22 β 977.48 / 987.68 Β±8.84 / 1002.03 ms β 977.16 / 987.63 Β±10.29 / 1006.44 ms β no change β
β QQuery 23 β 3000.63 / 3046.06 Β±37.37 / 3113.11 ms β 2958.54 / 3039.22 Β±88.25 / 3204.36 ms β no change β
β QQuery 24 β 40.79 / 44.59 Β±5.79 / 56.13 ms β 41.28 / 48.60 Β±13.44 / 75.48 ms β 1.09x slower β
β QQuery 25 β 109.83 / 114.20 Β±6.64 / 127.43 ms β 108.64 / 109.31 Β±0.56 / 110.33 ms β no change β
β QQuery 26 β 41.31 / 46.52 Β±7.65 / 61.73 ms β 41.13 / 47.43 Β±9.70 / 66.74 ms β no change β
β QQuery 27 β 508.08 / 516.37 Β±5.30 / 523.45 ms β 508.58 / 513.40 Β±3.44 / 518.91 ms β no change β
β QQuery 28 β 2907.16 / 2942.96 Β±35.07 / 3008.63 ms β 2980.89 / 3037.02 Β±49.41 / 3099.25 ms β no change β
β QQuery 29 β 41.41 / 52.36 Β±20.82 / 93.99 ms β 41.08 / 57.44 Β±31.57 / 120.56 ms β 1.10x slower β
β QQuery 30 β 302.75 / 307.19 Β±4.85 / 315.84 ms β 297.41 / 310.09 Β±8.62 / 322.16 ms β no change β
β QQuery 31 β 273.33 / 283.38 Β±6.78 / 293.89 ms β 280.18 / 290.74 Β±9.38 / 303.48 ms β no change β
β QQuery 32 β 3282.50 / 3445.59 Β±145.25 / 3620.75 ms β 3248.08 / 3336.39 Β±61.54 / 3412.53 ms β no change β
β QQuery 33 β 2590.55 / 2688.67 Β±62.72 / 2769.88 ms β 2520.60 / 2616.21 Β±99.98 / 2766.88 ms β no change β
β QQuery 34 β 2692.59 / 2721.40 Β±37.26 / 2794.25 ms β 2580.73 / 2649.07 Β±72.19 / 2784.12 ms β no change β
β QQuery 35 β 277.90 / 282.94 Β±5.46 / 292.69 ms β 300.20 / 333.61 Β±20.64 / 354.46 ms β 1.18x slower β
β QQuery 36 β 67.22 / 84.13 Β±27.76 / 139.47 ms β 70.81 / 73.49 Β±2.53 / 77.55 ms β +1.14x faster β
β QQuery 37 β 35.86 / 37.09 Β±1.37 / 39.54 ms β 38.17 / 38.51 Β±0.27 / 38.87 ms β no change β
β QQuery 38 β 40.22 / 41.81 Β±1.50 / 44.11 ms β 42.64 / 57.36 Β±24.04 / 105.23 ms β 1.37x slower β
β QQuery 39 β 130.00 / 140.95 Β±10.12 / 152.99 ms β 159.50 / 165.39 Β±5.39 / 173.60 ms β 1.17x slower β
β QQuery 40 β 13.96 / 14.43 Β±0.31 / 14.82 ms β 16.06 / 17.49 Β±1.13 / 19.06 ms β 1.21x slower β
β QQuery 41 β 13.59 / 15.05 Β±1.86 / 18.67 ms β 15.23 / 15.47 Β±0.17 / 15.75 ms β no change β
β QQuery 42 β 13.39 / 13.56 Β±0.24 / 14.03 ms β 14.62 / 14.81 Β±0.14 / 15.04 ms β 1.09x slower β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 27130.58ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 27179.87ms β
β Average Time (HEAD) β 630.94ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 632.09ms β
β Queries Faster β 4 β
β Queries Slower β 10 β
β Queries with No Change β 29 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `da89c7c (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 757.5 MiB | 772.8 MiB | +2.0% |
| Query 5 | 1.1 GiB | 1.2 GiB | +6.7% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 60.2 MiB | 50.3 MiB | -16.4% |
| Query 8 | 846.7 MiB | 899.0 MiB | +6.2% |
| Query 9 | 677.7 MiB | 593.5 MiB | -12.4% |
| Query 10 | 104.3 MiB | 112.8 MiB | +8.1% |
| Query 11 | 110.0 MiB | 114.7 MiB | +4.2% |
| Query 12 | 1.3 GiB | 1.3 GiB | -3.8% |
| Query 13 | 1.0 GiB | 1.0 GiB | +0.5% |
| Query 14 | 1.2 GiB | 1.3 GiB | +1.8% |
| Query 15 | 1.2 GiB | 1.2 GiB | +0.9% |
| Query 16 | 1.8 GiB | 1.8 GiB | -1.0% |
| Query 17 | 1.7 GiB | 2.1 GiB | +26.7% |
| Query 18 | 1.9 GiB | 1.8 GiB | -8.0% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.3 MiB | -1.2% |
| Query 22 | 3.0 MiB | 2.6 MiB | -13.8% |
| Query 23 | 29.3 MiB | 25.2 MiB | -13.9% |
| Query 24 | 60.8 MiB | 61.6 MiB | +1.2% |
| Query 25 | 172.1 MiB | 177.3 MiB | +3.0% |
| Query 26 | 63.8 MiB | 60.8 MiB | -4.7% |
| Query 27 | 2.4 MiB | 2.4 MiB | +0.0% |
| Query 28 | 1.5 GiB | 1.5 GiB | -4.8% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 731.0 MiB | 662.2 MiB | -9.4% |
| Query 31 | 1.5 GiB | 1.5 GiB | +0.0% |
| Query 32 | 928.7 MiB | 926.4 MiB | -0.3% |
| Query 33 | 2.1 GiB | 2.1 GiB | +1.9% |
| Query 34 | 2.1 GiB | 2.0 GiB | -4.4% |
| Query 35 | 596.1 MiB | 595.9 MiB | -0.0% |
| Query 36 | 122.5 MiB | 113.9 MiB | -7.0% |
| Query 37 | 6.9 MiB | 6.9 MiB | +0.0% |
| Query 38 | 5.2 MiB | 5.6 MiB | +8.5% |
| Query 39 | 297.8 MiB | 297.8 MiB | +0.0% |
| Query 40 | 1.7 MiB | 2.0 MiB | +17.2% |
| Query 41 | 3.1 MiB | 3.1 MiB | -0.0% |
| Query 42 | 1.7 MiB | 2.1 MiB | +25.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`da89c7c (merge-base)`) | 2.1 GiB | 9.3 GiB | 7.2 GiB | 4.4Γ |
| clickbench_partitioned | changed (`claude/single-distinct-to-groupby-allow-count`) | 2.1 GiB | 8.6 GiB | 6.5 GiB | 4.1Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 9.3 GiB |
| Avg memory | 5.6 GiB |
| CPU user | 1383.9s |
| CPU sys | 128.0s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 8.6 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1379.7s |
| CPU sys | 133.6s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Allow a non-distinct count alongside the single distinct aggregate in SingleDistinctToGroupBy [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #24859:
URL: https://github.com/apache/datafusion/pull/24859#issuecomment-5515048817
π€ Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/24859#issuecomment-5514755290)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing claude/single-distinct-to-groupby-allow-count (d510dd45f8ad9008f9312124913d7defa503910a) to da89c7c (merge-base) [diff](https://github.com/apache/datafusion/compare/da89c7c85b1c888ba4038317eacbdcfe03ab9b1a..d510dd45f8ad9008f9312124913d7defa503910a)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: "4G"
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 ms β 1.21 ms β no change β
β QQuery 1 β 11.84 ms β 11.85 ms β no change β
β QQuery 2 β 37.10 ms β 36.43 ms β no change β
β QQuery 3 β 31.32 ms β 31.30 ms β no change β
β QQuery 4 β 227.95 ms β 223.17 ms β no change β
β QQuery 5 β 278.44 ms β 273.19 ms β no change β
β QQuery 6 β 1.28 ms β 1.27 ms β no change β
β QQuery 7 β 13.60 ms β 13.41 ms β no change β
β QQuery 8 β 334.97 ms β 336.15 ms β no change β
β QQuery 9 β 462.84 ms β 455.82 ms β no change β
β QQuery 10 β 70.72 ms β 70.27 ms β no change β
β QQuery 11 β 82.51 ms β 81.12 ms β no change β
β QQuery 12 β 271.21 ms β 270.32 ms β no change β
β QQuery 13 β 990.69 ms β 971.51 ms β no change β
β QQuery 14 β 285.44 ms β 284.86 ms β no change β
β QQuery 15 β 271.87 ms β 263.20 ms β no change β
β QQuery 16 β 1224.39 ms β 1207.65 ms β no change β
β QQuery 17 β 938.17 ms β 928.40 ms β no change β
β QQuery 18 β 2525.15 ms β 2495.36 ms β no change β
β QQuery 19 β 28.34 ms β 27.90 ms β no change β
β QQuery 20 β 525.20 ms β 523.29 ms β no change β
β QQuery 21 β 517.57 ms β 515.53 ms β no change β
β QQuery 22 β 988.34 ms β 983.42 ms β no change β
β QQuery 23 β 3019.56 ms β 3020.27 ms β no change β
β QQuery 24 β 41.26 ms β 41.15 ms β no change β
β QQuery 25 β 109.96 ms β 110.13 ms β no change β
β QQuery 26 β 42.24 ms β 41.22 ms β no change β
β QQuery 27 β 512.82 ms β 522.42 ms β no change β
β QQuery 28 β 2944.09 ms β 2916.45 ms β no change β
β QQuery 29 β 41.01 ms β 41.20 ms β no change β
β QQuery 30 β 303.22 ms β 300.73 ms β no change β
β QQuery 31 β 289.32 ms β 282.18 ms β no change β
β QQuery 32 β 3284.67 ms β 3283.17 ms β no change β
β QQuery 33 β 2556.04 ms β 2592.71 ms β no change β
β QQuery 34 β 2630.88 ms β 2647.50 ms β no change β
β QQuery 35 β 338.93 ms β 284.82 ms β +1.19x faster β
β QQuery 36 β 69.03 ms β 64.59 ms β +1.07x faster β
β QQuery 37 β 36.77 ms β 35.69 ms β no change β
β QQuery 38 β 42.58 ms β 40.14 ms β +1.06x faster β
β QQuery 39 β 152.46 ms β 133.46 ms β +1.14x faster β
β QQuery 40 β 16.37 ms β 14.15 ms β +1.16x faster β
β QQuery 41 β 15.40 ms β 13.77 ms β +1.12x faster β
β QQuery 42 β 13.40 ms β 13.30 ms β no change β
βββββββββββββ΄βββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 26580.17ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 26405.69ms β
β Average Time (HEAD) β 618.14ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 614.09ms β
β Queries Faster β 6 β
β Queries Slower β 0 β
β Queries with No Change β 37 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
Distribution per query (min / mean Β±stddev / max):
Comparing HEAD and claude_single-distinct-to-groupby-allow-count
--------------------
Benchmark clickbench_partitioned.json
--------------------
βββββββββββββ³βββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββββββββββββββββββββββββββββββββββ³ββββββββββββββββ
β Query β HEAD β claude_single-distinct-to-groupby-allow-count β Change β
β‘βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β QQuery 0 β 1.22 / 4.03 Β±5.52 / 15.07 ms β 1.21 / 4.07 Β±5.62 / 15.31 ms β no change β
β QQuery 1 β 11.84 / 12.04 Β±0.13 / 12.20 ms β 11.85 / 12.04 Β±0.14 / 12.18 ms β no change β
β QQuery 2 β 37.10 / 37.32 Β±0.21 / 37.70 ms β 36.43 / 36.77 Β±0.27 / 37.14 ms β no change β
β QQuery 3 β 31.32 / 31.96 Β±0.75 / 33.37 ms β 31.30 / 32.04 Β±0.72 / 33.31 ms β no change β
β QQuery 4 β 227.95 / 230.00 Β±2.10 / 233.78 ms β 223.17 / 225.87 Β±1.68 / 227.65 ms β no change β
β QQuery 5 β 278.44 / 279.03 Β±0.47 / 279.59 ms β 273.19 / 277.34 Β±4.14 / 282.77 ms β no change β
β QQuery 6 β 1.28 / 1.42 Β±0.22 / 1.86 ms β 1.27 / 1.42 Β±0.23 / 1.88 ms β no change β
β QQuery 7 β 13.60 / 13.73 Β±0.09 / 13.86 ms β 13.41 / 14.00 Β±0.38 / 14.52 ms β no change β
β QQuery 8 β 334.97 / 339.76 Β±4.12 / 345.24 ms β 336.15 / 422.42 Β±164.81 / 751.89 ms β 1.24x slower β
β QQuery 9 β 462.84 / 473.37 Β±5.98 / 479.47 ms β 455.82 / 466.55 Β±9.97 / 482.30 ms β no change β
β QQuery 10 β 70.72 / 72.05 Β±1.05 / 73.56 ms β 70.27 / 74.76 Β±8.10 / 90.96 ms β no change β
β QQuery 11 β 82.51 / 85.86 Β±6.16 / 98.18 ms β 81.12 / 82.61 Β±1.13 / 83.95 ms β no change β
β QQuery 12 β 271.21 / 274.38 Β±3.28 / 279.68 ms β 270.32 / 274.99 Β±3.16 / 280.26 ms β no change β
β QQuery 13 β 990.69 / 997.89 Β±5.99 / 1005.99 ms β 971.51 / 994.19 Β±14.38 / 1009.62 ms β no change β
β QQuery 14 β 285.44 / 291.48 Β±3.21 / 294.20 ms β 284.86 / 288.27 Β±3.12 / 292.14 ms β no change β
β QQuery 15 β 271.87 / 274.95 Β±3.34 / 281.34 ms β 263.20 / 272.39 Β±8.06 / 287.09 ms β no change β
β QQuery 16 β 1224.39 / 1253.92 Β±19.39 / 1281.87 ms β 1207.65 / 1240.15 Β±23.07 / 1267.82 ms β no change β
β QQuery 17 β 938.17 / 964.06 Β±24.48 / 1003.91 ms β 928.40 / 937.91 Β±7.10 / 949.12 ms β no change β
β QQuery 18 β 2525.15 / 2581.99 Β±30.34 / 2607.28 ms β 2495.36 / 2605.08 Β±169.51 / 2941.99 ms β no change β
β QQuery 19 β 28.34 / 28.90 Β±0.58 / 29.94 ms β 27.90 / 28.46 Β±0.48 / 29.07 ms β no change β
β QQuery 20 β 525.20 / 536.89 Β±7.11 / 544.22 ms β 523.29 / 528.63 Β±7.76 / 543.93 ms β no change β
β QQuery 21 β 517.57 / 520.31 Β±1.95 / 522.69 ms β 515.53 / 523.04 Β±6.77 / 532.36 ms β no change β
β QQuery 22 β 988.34 / 1001.63 Β±12.13 / 1017.49 ms β 983.42 / 994.83 Β±12.13 / 1016.41 ms β no change β
β QQuery 23 β 3019.56 / 3035.98 Β±16.50 / 3065.28 ms β 3020.27 / 3049.11 Β±17.13 / 3071.25 ms β no change β
β QQuery 24 β 41.26 / 42.45 Β±1.80 / 46.03 ms β 41.15 / 42.19 Β±1.24 / 44.58 ms β no change β
β QQuery 25 β 109.96 / 112.39 Β±3.13 / 118.54 ms β 110.13 / 111.02 Β±0.57 / 111.82 ms β no change β
β QQuery 26 β 42.24 / 42.97 Β±0.69 / 43.93 ms β 41.22 / 41.58 Β±0.27 / 42.02 ms β no change β
β QQuery 27 β 512.82 / 526.02 Β±11.16 / 539.89 ms β 522.42 / 528.21 Β±4.27 / 533.77 ms β no change β
β QQuery 28 β 2944.09 / 2971.20 Β±20.57 / 3005.46 ms β 2916.45 / 2947.55 Β±27.99 / 2989.42 ms β no change β
β QQuery 29 β 41.01 / 43.94 Β±5.16 / 54.25 ms β 41.20 / 44.30 Β±4.62 / 53.34 ms β no change β
β QQuery 30 β 303.22 / 310.81 Β±9.03 / 326.79 ms β 300.73 / 309.74 Β±7.10 / 321.89 ms β no change β
β QQuery 31 β 289.32 / 298.20 Β±6.15 / 308.15 ms β 282.18 / 291.67 Β±11.42 / 308.58 ms β no change β
β QQuery 32 β 3284.67 / 3322.65 Β±27.89 / 3350.85 ms β 3283.17 / 3347.84 Β±43.18 / 3408.73 ms β no change β
β QQuery 33 β 2556.04 / 2627.15 Β±45.76 / 2678.81 ms β 2592.71 / 2646.43 Β±49.60 / 2715.34 ms β no change β
β QQuery 34 β 2630.88 / 2838.41 Β±161.17 / 3093.67 ms β 2647.50 / 2735.70 Β±59.70 / 2816.97 ms β no change β
β QQuery 35 β 338.93 / 354.55 Β±13.22 / 373.40 ms β 284.82 / 296.84 Β±9.47 / 307.76 ms β +1.19x faster β
β QQuery 36 β 69.03 / 76.47 Β±6.95 / 89.31 ms β 64.59 / 71.83 Β±5.20 / 77.62 ms β +1.06x faster β
β QQuery 37 β 36.77 / 37.43 Β±0.54 / 38.26 ms β 35.69 / 37.36 Β±1.53 / 39.68 ms β no change β
β QQuery 38 β 42.58 / 54.55 Β±21.43 / 97.34 ms β 40.14 / 40.46 Β±0.28 / 40.99 ms β +1.35x faster β
β QQuery 39 β 152.46 / 156.61 Β±3.70 / 162.37 ms β 133.46 / 154.18 Β±24.35 / 202.03 ms β no change β
β QQuery 40 β 16.37 / 17.84 Β±1.91 / 21.58 ms β 14.15 / 14.62 Β±0.32 / 15.05 ms β +1.22x faster β
β QQuery 41 β 15.40 / 15.89 Β±0.50 / 16.58 ms β 13.77 / 14.16 Β±0.31 / 14.61 ms β +1.12x faster β
β QQuery 42 β 13.40 / 13.91 Β±0.30 / 14.23 ms β 13.30 / 13.43 Β±0.10 / 13.54 ms β no change β
βββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββ΄ββββββββββββββββ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ³βββββββββββββ
β Benchmark Summary β β
β‘ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ©
β Total Time (HEAD) β 27206.37ms β
β Total Time (claude_single-distinct-to-groupby-allow-count) β 27076.07ms β
β Average Time (HEAD) β 632.71ms β
β Average Time (claude_single-distinct-to-groupby-allow-count) β 629.68ms β
β Queries Faster β 5 β
β Queries Slower β 1 β
β Queries with No Change β 37 β
β Queries with Failure β 0 β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββ
```
</p>
</details>
<details><summary>Memory Pool Peaks</summary>
Peak `MemoryPool` reservation per query β what DataFusion's accounting believes it reserved. Recorded only when the benchmark runs with `DATAFUSION_RUNTIME_MEMORY_LIMIT` set.
Base: `da89c7c (merge-base)` | Changed: `claude/single-distinct-to-groupby-allow-count`
**`clickbench_partitioned`**
| Query | Base | Changed | Change |
| --- | --- | --- | --- |
| Query 0 | 0 B | 0 B | 0.0% |
| Query 1 | 104 B | 104 B | +0.0% |
| Query 2 | 936 B | 936 B | +0.0% |
| Query 3 | 312 B | 312 B | +0.0% |
| Query 4 | 761.5 MiB | 758.3 MiB | -0.4% |
| Query 5 | 1.2 GiB | 1.1 GiB | -4.1% |
| Query 6 | 0 B | 0 B | 0.0% |
| Query 7 | 50.2 MiB | 70.1 MiB | +39.7% |
| Query 8 | 869.5 MiB | 883.8 MiB | +1.6% |
| Query 9 | 593.9 MiB | 551.7 MiB | -7.1% |
| Query 10 | 107.5 MiB | 114.9 MiB | +6.8% |
| Query 11 | 111.9 MiB | 117.8 MiB | +5.3% |
| Query 12 | 1.3 GiB | 1.3 GiB | +1.2% |
| Query 13 | 1018.6 MiB | 1014.4 MiB | -0.4% |
| Query 14 | 1.3 GiB | 1.3 GiB | -2.1% |
| Query 15 | 1.2 GiB | 1.2 GiB | -0.2% |
| Query 16 | 1.7 GiB | 2.0 GiB | +14.8% |
| Query 17 | 1.8 GiB | 1.9 GiB | +6.3% |
| Query 18 | 1.8 GiB | 1.7 GiB | -4.6% |
| Query 19 | 0 B | 0 B | 0.0% |
| Query 20 | 104 B | 104 B | +0.0% |
| Query 21 | 3.3 MiB | 3.3 MiB | -0.6% |
| Query 22 | 4.2 MiB | 3.1 MiB | -24.4% |
| Query 23 | 27.3 MiB | 28.4 MiB | +4.1% |
| Query 24 | 59.3 MiB | 60.8 MiB | +2.5% |
| Query 25 | 178.2 MiB | 177.9 MiB | -0.2% |
| Query 26 | 61.5 MiB | 60.2 MiB | -2.2% |
| Query 27 | 2.4 MiB | 2.2 MiB | -9.1% |
| Query 28 | 1.5 GiB | 1.5 GiB | +1.5% |
| Query 29 | 624 B | 624 B | +0.0% |
| Query 30 | 724.4 MiB | 733.8 MiB | +1.3% |
| Query 31 | 1.5 GiB | 1.5 GiB | -3.1% |
| Query 32 | 928.7 MiB | 927.5 MiB | -0.1% |
| Query 33 | 2.1 GiB | 2.1 GiB | -2.0% |
| Query 34 | 2.0 GiB | 2.2 GiB | +6.4% |
| Query 35 | 596.8 MiB | 593.9 MiB | -0.5% |
| Query 36 | 124.8 MiB | 117.7 MiB | -5.7% |
| Query 37 | 6.3 MiB | 6.9 MiB | +10.0% |
| Query 38 | 5.6 MiB | 5.7 MiB | +0.7% |
| Query 39 | 298.3 MiB | 297.8 MiB | -0.2% |
| Query 40 | 1.8 MiB | 1.7 MiB | -5.9% |
| Query 41 | 3.1 MiB | 3.1 MiB | +0.0% |
| Query 42 | 1.9 MiB | 1.6 MiB | -11.1% |
**Pool accounting vs. process RSS**
Max pool peak is the largest reservation any single query in the run reached; peak RSS covers the whole invocation, including data loading and allocator retention, and the two high-water marks need not coincide in time. The gap is therefore an upper bound on what the pool did not account for, not a measurement of it.
| Benchmark | Side | Max pool peak | Peak RSS | Gap | RSS / pool |
| --- | --- | --- | --- | --- | --- |
| clickbench_partitioned | base (`da89c7c (merge-base)`) | 2.1 GiB | 8.5 GiB | 6.4 GiB | 4.0Γ |
| clickbench_partitioned | changed (`claude/single-distinct-to-groupby-allow-count`) | 2.2 GiB | 8.7 GiB | 6.5 GiB | 4.0Γ |
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned β base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 8.5 GiB |
| Avg memory | 4.9 GiB |
| CPU user | 1369.2s |
| CPU sys | 136.3s |
| Peak spill | 0 B |
**clickbench_partitioned β branch**
| Metric | Value |
|--------|-------|
| Wall time | 140.0s |
| Peak memory | 8.7 GiB |
| Avg memory | 5.1 GiB |
| CPU user | 1369.4s |
| CPU sys | 135.5s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org