You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/09/13 01:14:24 UTC
[PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
jayzhan211 opened a new pull request, #25246:
URL: https://github.com/apache/datafusion/pull/25246
## Which issue does this PR close?
- Part of #24704 — the first step toward blocked group state, as discussed there. Related: #22712, #7065, #19562, #23178.
This branch carries three commits. The first (`perf: keep the key in the hash entry of GroupValuesPrimitive`) is independent of the other two and will be filed as its own small PR; it is included here because the blocked group values below rely on it to keep the probe off the value storage. The remaining two commits are the subject of this PR and are best read one at a time.
## Rationale for this change
### Commit 2 — blocked group state addressed by an independent resolve pass
Every attempt to store hash-aggregation state in blocks — so it can be emitted, freed and spilled one block at a time — has regressed at high cardinality: #22712, and 1.5–1.7x at 10M groups when retried for this work, for any block size. The reason is mechanical and is neither emission nor the block size: `blocks[g >> SHIFT][g & MASK]` is a **dependent load**. The block pointer has to resolve before the state access can issue, so the update loop keeps fewer cache misses in flight, and a flat `values[g]` loop wins on memory-level parallelism alone.
`intern` already produces the whole batch's group indices before any accumulator runs, so the addresses can be resolved in a separate, independent pass and the update loop can run through them. Because that pass knows every address without a dependent load, it can also prefetch each element — which does nothing in the naive loop, where the address itself is the dependency. Interleaved standalone measurement of the update loop (ns/row, M4 Pro, `i64` sum, uniformly random groups; the loop is bound by how many rows fit in the reorder window — even a bounds check is worth 27% here, which is why the resolve pass lives inside `accumulate` rather than in the per-row callback):
| groups | flat, unchecked (today) | naive blocked | resolve, no prefetch | **resolve + prefetch** |
|---|---|---|---|---|
| 1M | 0.87 | 1.18 | 1.09 | 1.11 (single block ⇒ today's loop is used) |
| 10M | 2.61 | 4.91 (1.9x) | 2.91 (1.11x) | **2.59 (0.99x)** |
| 20M | 3.21 | 5.36 | 3.55 | 3.35 (1.04x) |
| 40M | 3.60 | 5.57 | 3.92 | 3.71 (1.03x) |
Tables of up to 2^20 groups are one block and run exactly today's loop. With this in place, storing state in blocks costs nothing on the hot loop, and block-wise emission (commit 3) is an output-side change that cannot regress the update loop.
### Commit 3 — emit and free the output one block at a time
Hash aggregation materializes its whole output at once and holds it, fully reserved, until the last slice has been polled. When the consumer builds its own state from that output (a second aggregation, a join build side), the peak is *producer output + consumer state*, and under a memory limit the consumer spills even though the producer's memory is dead weight by then.
With the state in blocks, `EmitTo::First(BLOCK_LEN)` is a whole-block move with no renumbering of what remains, so the output side can hand over one block, slice it per poll, release it, and emit the next: the reservation shrinks block by block.
Nested aggregation (`GROUP BY k` over the output of `GROUP BY k`, 10M random groups, 40M rows, 1 partition, `datafusion-cli`, greedy pool), bytes spilled by the outer / inner aggregate at a fixed limit:
| limit | main | commit 2 only | **commits 2 + 3** |
|---|---|---|---|
| 640 MB | 310 MB / 1241 MB | 310 MB / 0 | **0 / 0** |
| 768 MB | 310 / 1241 | 310 / 0 | **0 / 0** |
| 896 MB | 310 / 0 | 0 / 0 | 0 / 0 |
The inner's 1241 → 0 is commit 2 (no doubling slack in the reservation); the outer's 310 → 0 is commit 3 (the inner's output is released while the outer builds). Put differently, the smallest limit at which this query runs without touching disk goes from over 1 GB to 640 MB. Note on arbitration: at exactly 512 MB under the greedy pool the inner's spill *replay* can be starved once the outer has grown into the pool (main survives there only because its coarser doubling OOMs the outer earlier); the fair pool passes at 448–576 MB and greedy passes from 528 MB — a pool-fairness boundary, not specific to this change.
## What changes are included in this PR?
Commit 2:
- `BlockedVec<T, SHIFT = 20>` (`functions-aggregate-common::groups_accumulator::blocks`): per-group state in blocks of 2^20; `as_single_block_mut` exposes a flat slice for the single-block case; `AddressResolver::resolve` is the independent pass (with prefetch); `take_all` / `take_first` / `get` / `resize` / `allocated_size`. Blocks grow by doubling, so only the last block carries slack.
- `accumulate_resolved` / `NullState::accumulate_resolved`: `accumulate` with a resolve callback per 256-row chunk; all four null/filter arms zip the resolved addresses with the data, so the update loop stays the lean one. `accumulate` itself is untouched.
- `PrimitiveGroupsAccumulator` (`SUM`/`MIN`/`MAX`/`BIT_*` on primitives) and `GroupValuesPrimitive` keep their values in a `BlockedVec`.
- `benches/blocked_update.rs`: flat vs naive blocked vs resolved at 100k/1M/10M, plus the real accumulator on random (seen-bit path) and first-appearance-dense (partial-stage path) indices.
Commit 3:
- `GroupsAccumulator::block_len() -> Option<usize>` (default `None`); `PrimitiveGroupsAccumulator` reports its block length.
- `GroupValues::block_len()` and `emit_block()` (defaults `None` / not implemented); `GroupValuesPrimitive::emit_block` moves the first block out and drops the hash index instead of renumbering it, since no `intern` follows.
- `AggregateHashTable::next_output_batch_inner`: new `OutputtingBlock` state; when every column agrees on a block length, output is emitted a block at a time and `memory_size()` is the remaining state plus the current block. Otherwise today's materialize-all path is used unchanged, so accumulators without block storage (`count`, `avg`, …) behave exactly as before. All three hash streams (partial, final, single) go through it.
Commit 1 (to be split out): `GroupValuesPrimitive` stores `(group_index, key)` instead of `(group_index, hash)`, so a probe never reads the values (0.90–0.93x on `GROUP BY l_partkey`); for natives wider than 8 bytes the entry grows to 32 bytes (+13% RSS measured on a Decimal128 key with 10M groups, at unchanged time).
No behavior change anywhere; the two new trait methods have defaults, `EmitTo` is unchanged.
## Measured against main
Binaries built in separate target dirs, interleaved runs:
- Real `PrimitiveGroupsAccumulator<Int64>::update_batch`: seen-bit path 1.00 / 1.00 / **0.90** at 100k / 1M / 10M groups; dense path 1.00 / **0.94** / 0.98–1.02 at 1M / 10M / 20M.
- `datafusion-cli`: single-block queries 0.99–1.03x; 10M-group queries 1.00–1.02x with max RSS 6–16% lower; nullable input 0.99x, `FILTER` 1.02x, Float64 keys 1.01x, `count`+`avg` (old output path) 0.96x; SF10 `l_orderkey` (15M clustered groups) 0.98x with RSS 4.7 → 3.8 GB; SF10 `l_partkey` (2M random) 0.90x.
- TPC-H SF10 (`dfbench tpch`, 4 runs, min per query): total **0.98x**; q17 0.91x; q18 (`GROUP BY l_orderkey`, ~1.25M groups per partition, the two-block band) +2.3%, the one recurring small regression.
- Not run: ClickBench (no `hits.parquet` locally); it is the suite where high-cardinality primitive `GROUP BY` shows most, so a run there would be welcome.
## What is the testing strategy for this PR?
- `blocks::tests`: indexing/resize across blocks, `take_all`/`take_first` order and packing, whole-block moves, growth by doubling, resolver addresses.
- `accumulate::test::accumulate_resolved_matches_accumulate`: same rows in the same order as `accumulate` for every null/filter combination across chunk boundaries.
- `prim_op::tests::multi_block_groups_update_and_emit_in_order`, `primitive::tests::intern_and_emit_across_blocks`, `primitive::tests::emit_block_walks_the_blocks_in_order`.
- `hash_stream::tests::test_single_hash_stream_releases_state_block_by_block`: two blocks of `sum(Int64)`; group order preserved across the block boundary, the reservation while the second block is sliced is < 1/4 of what it was during the first, and 0 at the end.
- Extended test suite and full sqllogictest suite green.
## Are there any user-facing changes?
No. New default trait methods on `GroupsAccumulator` and `GroupValues`; existing implementors compile and behave unchanged.
https://claude.ai/code/session_01N4Hye6EyCKWzNDjyoBAozd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654162250
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and blocked-emit
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 5.70 ms │ 5.84 ms │ no change │
│ QQuery 2 │ 80.67 ms │ 79.65 ms │ no change │
│ QQuery 3 │ 29.69 ms │ 28.97 ms │ no change │
│ QQuery 4 │ 515.93 ms │ 508.66 ms │ no change │
│ QQuery 5 │ 52.27 ms │ 52.26 ms │ no change │
│ QQuery 6 │ 36.29 ms │ 36.56 ms │ no change │
│ QQuery 7 │ 74.46 ms │ 74.20 ms │ no change │
│ QQuery 8 │ 37.65 ms │ 37.00 ms │ no change │
│ QQuery 9 │ 54.86 ms │ 50.38 ms │ +1.09x faster │
│ QQuery 10 │ 63.23 ms │ 63.14 ms │ no change │
│ QQuery 11 │ 319.31 ms │ 318.96 ms │ no change │
│ QQuery 12 │ 29.33 ms │ 29.33 ms │ no change │
│ QQuery 13 │ 119.06 ms │ 117.59 ms │ no change │
│ QQuery 14 │ 421.62 ms │ 417.55 ms │ no change │
│ QQuery 15 │ 61.88 ms │ 57.16 ms │ +1.08x faster │
│ QQuery 16 │ 7.39 ms │ 6.88 ms │ +1.07x faster │
│ QQuery 17 │ 80.48 ms │ 81.44 ms │ no change │
│ QQuery 18 │ 106.03 ms │ 105.30 ms │ no change │
│ QQuery 19 │ 42.21 ms │ 41.15 ms │ no change │
│ QQuery 20 │ 36.48 ms │ 36.06 ms │ no change │
│ QQuery 21 │ 18.02 ms │ 17.81 ms │ no change │
│ QQuery 22 │ 66.10 ms │ 66.30 ms │ no change │
│ QQuery 23 │ 313.67 ms │ 316.29 ms │ no change │
│ QQuery 24 │ 203.18 ms │ 200.76 ms │ no change │
│ QQuery 25 │ 109.38 ms │ 111.08 ms │ no change │
│ QQuery 26 │ 49.97 ms │ 48.17 ms │ no change │
│ QQuery 27 │ 6.03 ms │ 6.13 ms │ no change │
│ QQuery 28 │ 57.79 ms │ 59.88 ms │ no change │
│ QQuery 29 │ 97.33 ms │ 97.21 ms │ no change │
│ QQuery 30 │ 33.69 ms │ 32.41 ms │ no change │
│ QQuery 31 │ 111.24 ms │ 112.08 ms │ no change │
│ QQuery 32 │ 20.62 ms │ 21.48 ms │ no change │
│ QQuery 33 │ 38.05 ms │ 37.77 ms │ no change │
│ QQuery 34 │ 10.48 ms │ 9.97 ms │ no change │
│ QQuery 35 │ 77.48 ms │ 73.21 ms │ +1.06x faster │
│ QQuery 36 │ 5.93 ms │ 5.79 ms │ no change │
│ QQuery 37 │ 7.15 ms │ 6.80 ms │ no change │
│ QQuery 38 │ 62.62 ms │ 62.39 ms │ no change │
│ QQuery 39 │ 90.73 ms │ 89.54 ms │ no change │
│ QQuery 40 │ 25.77 ms │ 24.58 ms │ no change │
│ QQuery 41 │ 11.25 ms │ 11.89 ms │ 1.06x slower │
│ QQuery 42 │ 24.26 ms │ 24.08 ms │ no change │
│ QQuery 43 │ 5.10 ms │ 5.04 ms │ no change │
│ QQuery 44 │ 9.55 ms │ 9.33 ms │ no change │
│ QQuery 45 │ 40.62 ms │ 41.06 ms │ no change │
│ QQuery 46 │ 12.15 ms │ 12.42 ms │ no change │
│ QQuery 47 │ 236.61 ms │ 236.61 ms │ no change │
│ QQuery 48 │ 98.26 ms │ 96.17 ms │ no change │
│ QQuery 49 │ 71.48 ms │ 72.55 ms │ no change │
│ QQuery 50 │ 59.87 ms │ 60.06 ms │ no change │
│ QQuery 51 │ 92.27 ms │ 93.95 ms │ no change │
│ QQuery 52 │ 24.72 ms │ 24.62 ms │ no change │
│ QQuery 53 │ 29.92 ms │ 29.45 ms │ no change │
│ QQuery 54 │ 54.86 ms │ 54.96 ms │ no change │
│ QQuery 55 │ 24.76 ms │ 24.82 ms │ no change │
│ QQuery 56 │ 39.76 ms │ 39.44 ms │ no change │
│ QQuery 57 │ 176.49 ms │ 177.50 ms │ no change │
│ QQuery 58 │ 112.96 ms │ 111.56 ms │ no change │
│ QQuery 59 │ 117.73 ms │ 119.19 ms │ no change │
│ QQuery 60 │ 38.80 ms │ 39.57 ms │ no change │
│ QQuery 61 │ 12.24 ms │ 12.55 ms │ no change │
│ QQuery 62 │ 46.06 ms │ 46.11 ms │ no change │
│ QQuery 63 │ 29.85 ms │ 29.80 ms │ no change │
│ QQuery 64 │ 369.40 ms │ 370.99 ms │ no change │
│ QQuery 65 │ 120.24 ms │ 124.60 ms │ no change │
│ QQuery 66 │ 79.81 ms │ 80.26 ms │ no change │
│ QQuery 67 │ 251.12 ms │ 252.30 ms │ no change │
│ QQuery 68 │ 11.94 ms │ 12.14 ms │ no change │
│ QQuery 69 │ 57.67 ms │ 57.48 ms │ no change │
│ QQuery 70 │ 104.69 ms │ 104.72 ms │ no change │
│ QQuery 71 │ 36.27 ms │ 35.80 ms │ no change │
│ QQuery 72 │ 1768.78 ms │ 1858.68 ms │ 1.05x slower │
│ QQuery 73 │ 10.34 ms │ 9.95 ms │ no change │
│ QQuery 74 │ 178.39 ms │ 183.21 ms │ no change │
│ QQuery 75 │ 146.47 ms │ 150.94 ms │ no change │
│ QQuery 76 │ 35.09 ms │ 35.72 ms │ no change │
│ QQuery 77 │ 61.29 ms │ 62.80 ms │ no change │
│ QQuery 78 │ 226.60 ms │ 228.64 ms │ no change │
│ QQuery 79 │ 66.73 ms │ 68.32 ms │ no change │
│ QQuery 80 │ 101.55 ms │ 100.65 ms │ no change │
│ QQuery 81 │ 26.22 ms │ 26.38 ms │ no change │
│ QQuery 82 │ 16.38 ms │ 16.68 ms │ no change │
│ QQuery 83 │ 34.63 ms │ 34.93 ms │ no change │
│ QQuery 84 │ 29.34 ms │ 29.60 ms │ no change │
│ QQuery 85 │ 103.37 ms │ 102.24 ms │ no change │
│ QQuery 86 │ 25.43 ms │ 25.45 ms │ no change │
│ QQuery 87 │ 61.49 ms │ 63.59 ms │ no change │
│ QQuery 88 │ 62.97 ms │ 64.26 ms │ no change │
│ QQuery 89 │ 36.15 ms │ 36.14 ms │ no change │
│ QQuery 90 │ 17.14 ms │ 17.24 ms │ no change │
│ QQuery 91 │ 44.76 ms │ 44.88 ms │ no change │
│ QQuery 92 │ 29.38 ms │ 29.67 ms │ no change │
│ QQuery 93 │ 49.31 ms │ 49.67 ms │ no change │
│ QQuery 94 │ 38.40 ms │ 39.98 ms │ no change │
│ QQuery 95 │ 80.43 ms │ 81.15 ms │ no change │
│ QQuery 96 │ 24.10 ms │ 24.10 ms │ no change │
│ QQuery 97 │ 51.73 ms │ 51.50 ms │ no change │
│ QQuery 98 │ 42.88 ms │ 43.19 ms │ no change │
│ QQuery 99 │ 69.62 ms │ 69.42 ms │ no change │
└───────────┴────────────┴──────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 9419.47ms │
│ Total Time (blocked-emit) │ 9507.75ms │
│ Average Time (HEAD) │ 95.15ms │
│ Average Time (blocked-emit) │ 96.04ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 2 │
│ Queries with No Change │ 93 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴───────────┘
Distribution per query (min / mean ±stddev / max):
Comparing HEAD and blocked-emit
--------------------
Benchmark tpcds_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 5.70 / 6.27 ±0.88 / 8.01 ms │ 5.84 / 6.40 ±0.90 / 8.18 ms │ no change │
│ QQuery 2 │ 80.67 / 81.28 ±0.37 / 81.61 ms │ 79.65 / 80.52 ±0.64 / 81.58 ms │ no change │
│ QQuery 3 │ 29.69 / 29.84 ±0.15 / 30.04 ms │ 28.97 / 29.16 ±0.17 / 29.38 ms │ no change │
│ QQuery 4 │ 515.93 / 519.83 ±4.34 / 525.74 ms │ 508.66 / 514.95 ±4.45 / 522.58 ms │ no change │
│ QQuery 5 │ 52.27 / 53.24 ±0.82 / 54.52 ms │ 52.26 / 52.81 ±0.35 / 53.29 ms │ no change │
│ QQuery 6 │ 36.29 / 37.02 ±0.53 / 37.94 ms │ 36.56 / 37.47 ±0.58 / 38.00 ms │ no change │
│ QQuery 7 │ 74.46 / 75.63 ±0.72 / 76.34 ms │ 74.20 / 75.49 ±1.08 / 77.09 ms │ no change │
│ QQuery 8 │ 37.65 / 39.03 ±2.04 / 43.06 ms │ 37.00 / 37.27 ±0.37 / 37.96 ms │ no change │
│ QQuery 9 │ 54.86 / 55.72 ±0.78 / 56.61 ms │ 50.38 / 53.98 ±2.31 / 56.97 ms │ no change │
│ QQuery 10 │ 63.23 / 63.62 ±0.25 / 63.99 ms │ 63.14 / 63.69 ±0.59 / 64.73 ms │ no change │
│ QQuery 11 │ 319.31 / 324.50 ±5.45 / 334.25 ms │ 318.96 / 323.41 ±3.64 / 327.81 ms │ no change │
│ QQuery 12 │ 29.33 / 29.68 ±0.30 / 30.16 ms │ 29.33 / 29.92 ±0.40 / 30.55 ms │ no change │
│ QQuery 13 │ 119.06 / 120.33 ±1.12 / 121.79 ms │ 117.59 / 119.81 ±1.89 / 123.15 ms │ no change │
│ QQuery 14 │ 421.62 / 424.31 ±1.68 / 426.29 ms │ 417.55 / 421.97 ±5.07 / 431.82 ms │ no change │
│ QQuery 15 │ 61.88 / 62.68 ±0.89 / 64.43 ms │ 57.16 / 59.75 ±1.92 / 61.95 ms │ no change │
│ QQuery 16 │ 7.39 / 7.49 ±0.06 / 7.59 ms │ 6.88 / 7.20 ±0.17 / 7.35 ms │ no change │
│ QQuery 17 │ 80.48 / 81.86 ±1.00 / 83.32 ms │ 81.44 / 83.39 ±2.89 / 89.11 ms │ no change │
│ QQuery 18 │ 106.03 / 107.96 ±1.58 / 109.65 ms │ 105.30 / 107.18 ±1.56 / 109.24 ms │ no change │
│ QQuery 19 │ 42.21 / 42.36 ±0.16 / 42.66 ms │ 41.15 / 41.84 ±0.48 / 42.55 ms │ no change │
│ QQuery 20 │ 36.48 / 38.18 ±1.54 / 41.08 ms │ 36.06 / 37.50 ±1.93 / 41.31 ms │ no change │
│ QQuery 21 │ 18.02 / 18.31 ±0.44 / 19.18 ms │ 17.81 / 17.92 ±0.11 / 18.11 ms │ no change │
│ QQuery 22 │ 66.10 / 66.92 ±0.85 / 68.22 ms │ 66.30 / 67.41 ±0.94 / 68.67 ms │ no change │
│ QQuery 23 │ 313.67 / 327.99 ±7.87 / 335.86 ms │ 316.29 / 322.55 ±7.11 / 331.98 ms │ no change │
│ QQuery 24 │ 203.18 / 210.26 ±4.73 / 216.74 ms │ 200.76 / 204.65 ±3.58 / 210.20 ms │ no change │
│ QQuery 25 │ 109.38 / 111.24 ±2.02 / 114.76 ms │ 111.08 / 113.22 ±2.35 / 117.67 ms │ no change │
│ QQuery 26 │ 49.97 / 51.75 ±2.66 / 57.03 ms │ 48.17 / 48.98 ±0.48 / 49.40 ms │ +1.06x faster │
│ QQuery 27 │ 6.03 / 6.23 ±0.21 / 6.62 ms │ 6.13 / 6.40 ±0.24 / 6.77 ms │ no change │
│ QQuery 28 │ 57.79 / 60.50 ±1.84 / 62.42 ms │ 59.88 / 61.72 ±1.60 / 64.68 ms │ no change │
│ QQuery 29 │ 97.33 / 98.32 ±1.21 / 100.65 ms │ 97.21 / 98.38 ±0.70 / 99.10 ms │ no change │
│ QQuery 30 │ 33.69 / 35.73 ±3.69 / 43.10 ms │ 32.41 / 32.78 ±0.26 / 33.15 ms │ +1.09x faster │
│ QQuery 31 │ 111.24 / 113.01 ±1.33 / 115.33 ms │ 112.08 / 114.23 ±1.75 / 116.84 ms │ no change │
│ QQuery 32 │ 20.62 / 21.13 ±0.35 / 21.44 ms │ 21.48 / 21.73 ±0.20 / 22.09 ms │ no change │
│ QQuery 33 │ 38.05 / 38.51 ±0.29 / 38.83 ms │ 37.77 / 38.81 ±0.76 / 40.00 ms │ no change │
│ QQuery 34 │ 10.48 / 10.68 ±0.21 / 11.00 ms │ 9.97 / 10.19 ±0.23 / 10.63 ms │ no change │
│ QQuery 35 │ 77.48 / 78.82 ±1.22 / 80.67 ms │ 73.21 / 78.29 ±3.33 / 82.66 ms │ no change │
│ QQuery 36 │ 5.93 / 6.09 ±0.13 / 6.33 ms │ 5.79 / 6.08 ±0.21 / 6.41 ms │ no change │
│ QQuery 37 │ 7.15 / 7.28 ±0.12 / 7.49 ms │ 6.80 / 7.02 ±0.13 / 7.14 ms │ no change │
│ QQuery 38 │ 62.62 / 63.77 ±1.07 / 65.20 ms │ 62.39 / 64.52 ±1.39 / 66.40 ms │ no change │
│ QQuery 39 │ 90.73 / 95.09 ±4.13 / 102.73 ms │ 89.54 / 94.06 ±5.98 / 105.88 ms │ no change │
│ QQuery 40 │ 25.77 / 26.58 ±0.84 / 28.15 ms │ 24.58 / 25.29 ±0.71 / 26.55 ms │ no change │
│ QQuery 41 │ 11.25 / 11.59 ±0.20 / 11.83 ms │ 11.89 / 12.04 ±0.17 / 12.37 ms │ no change │
│ QQuery 42 │ 24.26 / 24.56 ±0.26 / 24.93 ms │ 24.08 / 24.71 ±0.77 / 26.15 ms │ no change │
│ QQuery 43 │ 5.10 / 5.33 ±0.22 / 5.75 ms │ 5.04 / 5.20 ±0.26 / 5.72 ms │ no change │
│ QQuery 44 │ 9.55 / 9.73 ±0.10 / 9.85 ms │ 9.33 / 9.76 ±0.32 / 10.18 ms │ no change │
│ QQuery 45 │ 40.62 / 41.66 ±0.90 / 43.13 ms │ 41.06 / 42.51 ±1.21 / 44.32 ms │ no change │
│ QQuery 46 │ 12.15 / 12.39 ±0.17 / 12.58 ms │ 12.42 / 12.69 ±0.25 / 13.14 ms │ no change │
│ QQuery 47 │ 236.61 / 247.34 ±8.46 / 258.12 ms │ 236.61 / 245.23 ±8.23 / 259.02 ms │ no change │
│ QQuery 48 │ 98.26 / 99.27 ±0.72 / 100.34 ms │ 96.17 / 98.30 ±1.40 / 100.59 ms │ no change │
│ QQuery 49 │ 71.48 / 72.84 ±0.82 / 73.78 ms │ 72.55 / 72.71 ±0.12 / 72.86 ms │ no change │
│ QQuery 50 │ 59.87 / 63.31 ±5.54 / 74.35 ms │ 60.06 / 62.49 ±3.59 / 69.59 ms │ no change │
│ QQuery 51 │ 92.27 / 95.64 ±2.00 / 98.40 ms │ 93.95 / 95.70 ±1.01 / 96.80 ms │ no change │
│ QQuery 52 │ 24.72 / 24.88 ±0.12 / 25.08 ms │ 24.62 / 25.07 ±0.37 / 25.54 ms │ no change │
│ QQuery 53 │ 29.92 / 30.31 ±0.25 / 30.69 ms │ 29.45 / 29.92 ±0.28 / 30.23 ms │ no change │
│ QQuery 54 │ 54.86 / 58.22 ±4.90 / 67.95 ms │ 54.96 / 57.58 ±2.92 / 63.12 ms │ no change │
│ QQuery 55 │ 24.76 / 24.91 ±0.15 / 25.12 ms │ 24.82 / 25.74 ±0.83 / 27.28 ms │ no change │
│ QQuery 56 │ 39.76 / 40.15 ±0.31 / 40.68 ms │ 39.44 / 40.19 ±0.53 / 40.93 ms │ no change │
│ QQuery 57 │ 176.49 / 181.52 ±6.19 / 193.64 ms │ 177.50 / 179.86 ±1.36 / 181.28 ms │ no change │
│ QQuery 58 │ 112.96 / 115.21 ±3.60 / 122.28 ms │ 111.56 / 113.80 ±2.05 / 117.66 ms │ no change │
│ QQuery 59 │ 117.73 / 119.43 ±0.99 / 120.33 ms │ 119.19 / 120.47 ±1.39 / 123.18 ms │ no change │
│ QQuery 60 │ 38.80 / 39.93 ±0.91 / 41.35 ms │ 39.57 / 39.91 ±0.33 / 40.35 ms │ no change │
│ QQuery 61 │ 12.24 / 12.60 ±0.19 / 12.75 ms │ 12.55 / 12.84 ±0.19 / 13.06 ms │ no change │
│ QQuery 62 │ 46.06 / 48.52 ±2.75 / 53.71 ms │ 46.11 / 47.98 ±2.46 / 52.73 ms │ no change │
│ QQuery 63 │ 29.85 / 30.64 ±0.52 / 31.41 ms │ 29.80 / 30.59 ±0.87 / 32.28 ms │ no change │
│ QQuery 64 │ 369.40 / 375.44 ±3.95 / 380.95 ms │ 370.99 / 379.10 ±6.39 / 389.93 ms │ no change │
│ QQuery 65 │ 120.24 / 122.41 ±2.06 / 125.85 ms │ 124.60 / 127.39 ±2.29 / 130.21 ms │ no change │
│ QQuery 66 │ 79.81 / 82.83 ±2.36 / 85.57 ms │ 80.26 / 82.90 ±2.86 / 88.48 ms │ no change │
│ QQuery 67 │ 251.12 / 257.49 ±4.06 / 262.63 ms │ 252.30 / 259.78 ±4.43 / 264.78 ms │ no change │
│ QQuery 68 │ 11.94 / 12.25 ±0.21 / 12.51 ms │ 12.14 / 12.40 ±0.20 / 12.75 ms │ no change │
│ QQuery 69 │ 57.67 / 58.29 ±0.40 / 58.91 ms │ 57.48 / 58.43 ±0.63 / 59.42 ms │ no change │
│ QQuery 70 │ 104.69 / 107.38 ±3.20 / 113.42 ms │ 104.72 / 109.24 ±3.02 / 112.66 ms │ no change │
│ QQuery 71 │ 36.27 / 37.78 ±2.35 / 42.45 ms │ 35.80 / 36.21 ±0.34 / 36.81 ms │ no change │
│ QQuery 72 │ 1768.78 / 1896.46 ±64.61 / 1946.19 ms │ 1858.68 / 1901.86 ±47.93 / 1976.55 ms │ no change │
│ QQuery 73 │ 10.34 / 11.11 ±0.44 / 11.62 ms │ 9.95 / 12.15 ±3.40 / 18.92 ms │ 1.09x slower │
│ QQuery 74 │ 178.39 / 183.99 ±3.52 / 189.39 ms │ 183.21 / 189.61 ±3.51 / 193.99 ms │ no change │
│ QQuery 75 │ 146.47 / 151.20 ±4.65 / 159.74 ms │ 150.94 / 151.72 ±0.80 / 153.13 ms │ no change │
│ QQuery 76 │ 35.09 / 35.55 ±0.30 / 35.97 ms │ 35.72 / 36.46 ±0.55 / 37.43 ms │ no change │
│ QQuery 77 │ 61.29 / 62.72 ±0.87 / 63.81 ms │ 62.80 / 64.48 ±1.61 / 67.31 ms │ no change │
│ QQuery 78 │ 226.60 / 233.65 ±6.36 / 244.88 ms │ 228.64 / 235.98 ±5.21 / 242.39 ms │ no change │
│ QQuery 79 │ 66.73 / 67.40 ±0.50 / 68.23 ms │ 68.32 / 68.90 ±0.59 / 69.97 ms │ no change │
│ QQuery 80 │ 101.55 / 104.31 ±2.41 / 107.94 ms │ 100.65 / 103.39 ±2.52 / 107.80 ms │ no change │
│ QQuery 81 │ 26.22 / 26.36 ±0.08 / 26.44 ms │ 26.38 / 26.96 ±0.53 / 27.87 ms │ no change │
│ QQuery 82 │ 16.38 / 17.09 ±0.40 / 17.41 ms │ 16.68 / 17.15 ±0.43 / 17.96 ms │ no change │
│ QQuery 83 │ 34.63 / 34.91 ±0.25 / 35.21 ms │ 34.93 / 35.98 ±0.72 / 37.17 ms │ no change │
│ QQuery 84 │ 29.34 / 29.86 ±0.39 / 30.35 ms │ 29.60 / 29.92 ±0.35 / 30.54 ms │ no change │
│ QQuery 85 │ 103.37 / 108.00 ±4.83 / 116.51 ms │ 102.24 / 105.79 ±2.77 / 110.54 ms │ no change │
│ QQuery 86 │ 25.43 / 25.96 ±0.60 / 27.04 ms │ 25.45 / 25.72 ±0.27 / 26.21 ms │ no change │
│ QQuery 87 │ 61.49 / 63.99 ±1.80 / 66.17 ms │ 63.59 / 64.91 ±1.61 / 67.99 ms │ no change │
│ QQuery 88 │ 62.97 / 66.49 ±4.82 / 76.04 ms │ 64.26 / 64.72 ±0.57 / 65.82 ms │ no change │
│ QQuery 89 │ 36.15 / 36.85 ±0.44 / 37.48 ms │ 36.14 / 36.98 ±0.76 / 38.17 ms │ no change │
│ QQuery 90 │ 17.14 / 17.39 ±0.23 / 17.76 ms │ 17.24 / 17.46 ±0.18 / 17.66 ms │ no change │
│ QQuery 91 │ 44.76 / 45.83 ±0.68 / 46.74 ms │ 44.88 / 45.50 ±0.38 / 46.00 ms │ no change │
│ QQuery 92 │ 29.38 / 30.04 ±0.86 / 31.60 ms │ 29.67 / 30.77 ±0.71 / 31.80 ms │ no change │
│ QQuery 93 │ 49.31 / 50.40 ±0.69 / 51.24 ms │ 49.67 / 50.68 ±0.90 / 51.81 ms │ no change │
│ QQuery 94 │ 38.40 / 41.20 ±2.45 / 45.17 ms │ 39.98 / 41.78 ±2.05 / 45.40 ms │ no change │
│ QQuery 95 │ 80.43 / 81.99 ±1.15 / 83.86 ms │ 81.15 / 82.26 ±1.21 / 84.59 ms │ no change │
│ QQuery 96 │ 24.10 / 24.46 ±0.25 / 24.83 ms │ 24.10 / 24.40 ±0.26 / 24.81 ms │ no change │
│ QQuery 97 │ 51.73 / 52.76 ±1.14 / 54.31 ms │ 51.50 / 52.70 ±0.95 / 54.41 ms │ no change │
│ QQuery 98 │ 42.88 / 44.42 ±1.87 / 47.28 ms │ 43.19 / 44.45 ±0.96 / 46.10 ms │ no change │
│ QQuery 99 │ 69.62 / 70.26 ±0.43 / 70.76 ms │ 69.42 / 70.41 ±0.65 / 71.37 ms │ no change │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD) │ 9727.55ms │
│ Total Time (blocked-emit) │ 9721.82ms │
│ Average Time (HEAD) │ 98.26ms │
│ Average Time (blocked-emit) │ 98.20ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 1 │
│ Queries with No Change │ 96 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴───────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpcds — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.2 GiB |
| Avg memory | 1.5 GiB |
| CPU user | 208.4s |
| CPU sys | 5.9s |
| Peak spill | 0 B |
**tpcds — branch**
| Metric | Value |
|--------|-------|
| Wall time | 50.0s |
| Peak memory | 2.4 GiB |
| Avg memory | 1.6 GiB |
| CPU user | 210.3s |
| CPU sys | 5.8s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654181011
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and blocked-emit
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.40 ms │ 1.24 ms │ +1.13x faster │
│ QQuery 1 │ 12.21 ms │ 11.90 ms │ no change │
│ QQuery 2 │ 37.97 ms │ 37.77 ms │ no change │
│ QQuery 3 │ 32.79 ms │ 30.76 ms │ +1.07x faster │
│ QQuery 4 │ 269.05 ms │ 217.35 ms │ +1.24x faster │
│ QQuery 5 │ 274.65 ms │ 272.53 ms │ no change │
│ QQuery 6 │ 1.26 ms │ 1.30 ms │ no change │
│ QQuery 7 │ 13.13 ms │ 13.19 ms │ no change │
│ QQuery 8 │ 331.27 ms │ 326.08 ms │ no change │
│ QQuery 9 │ 465.04 ms │ 453.24 ms │ no change │
│ QQuery 10 │ 71.43 ms │ 69.37 ms │ no change │
│ QQuery 11 │ 81.47 ms │ 80.52 ms │ no change │
│ QQuery 12 │ 268.42 ms │ 265.57 ms │ no change │
│ QQuery 13 │ 414.77 ms │ 390.92 ms │ +1.06x faster │
│ QQuery 14 │ 303.52 ms │ 322.59 ms │ 1.06x slower │
│ QQuery 15 │ 284.17 ms │ 326.14 ms │ 1.15x slower │
│ QQuery 16 │ 622.30 ms │ 718.09 ms │ 1.15x slower │
│ QQuery 17 │ 621.09 ms │ 634.22 ms │ no change │
│ QQuery 18 │ 1264.07 ms │ 1264.55 ms │ no change │
│ QQuery 19 │ 27.38 ms │ 28.74 ms │ no change │
│ QQuery 20 │ 517.30 ms │ 519.49 ms │ no change │
│ QQuery 21 │ 516.92 ms │ 523.74 ms │ no change │
│ QQuery 22 │ 998.75 ms │ 982.57 ms │ no change │
│ QQuery 23 │ 3087.59 ms │ 3055.93 ms │ no change │
│ QQuery 24 │ 40.94 ms │ 40.97 ms │ no change │
│ QQuery 25 │ 110.27 ms │ 109.63 ms │ no change │
│ QQuery 26 │ 41.50 ms │ 43.89 ms │ 1.06x slower │
│ QQuery 27 │ 515.13 ms │ 517.80 ms │ no change │
│ QQuery 28 │ 2981.23 ms │ 2895.42 ms │ no change │
│ QQuery 29 │ 41.29 ms │ 43.79 ms │ 1.06x slower │
│ QQuery 30 │ 317.14 ms │ 360.25 ms │ 1.14x slower │
│ QQuery 31 │ 290.61 ms │ 315.30 ms │ 1.08x slower │
│ QQuery 32 │ 960.95 ms │ 972.71 ms │ no change │
│ QQuery 33 │ 1480.69 ms │ 1503.10 ms │ no change │
│ QQuery 34 │ 1475.04 ms │ 1485.80 ms │ no change │
│ QQuery 35 │ 274.67 ms │ 294.33 ms │ 1.07x slower │
│ QQuery 36 │ 72.73 ms │ 75.55 ms │ no change │
│ QQuery 37 │ 35.35 ms │ 38.55 ms │ 1.09x slower │
│ QQuery 38 │ 41.61 ms │ 41.16 ms │ no change │
│ QQuery 39 │ 144.92 ms │ 133.85 ms │ +1.08x faster │
│ QQuery 40 │ 14.51 ms │ 14.10 ms │ no change │
│ QQuery 41 │ 14.63 ms │ 13.51 ms │ +1.08x faster │
│ QQuery 42 │ 14.90 ms │ 12.68 ms │ +1.18x faster │
└───────────┴────────────┴──────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19386.09ms │
│ Total Time (blocked-emit) │ 19460.18ms │
│ Average Time (HEAD) │ 450.84ms │
│ Average Time (blocked-emit) │ 452.56ms │
│ Queries Faster │ 7 │
│ Queries Slower │ 9 │
│ Queries with No Change │ 27 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴────────────┘
Distribution per query (min / mean ±stddev / max):
Comparing HEAD and blocked-emit
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0 │ 1.40 / 4.45 ±5.86 / 16.16 ms │ 1.24 / 3.94 ±5.37 / 14.68 ms │ +1.13x faster │
│ QQuery 1 │ 12.21 / 12.76 ±0.30 / 13.11 ms │ 11.90 / 12.15 ±0.20 / 12.46 ms │ no change │
│ QQuery 2 │ 37.97 / 38.45 ±0.29 / 38.82 ms │ 37.77 / 38.15 ±0.21 / 38.38 ms │ no change │
│ QQuery 3 │ 32.79 / 33.55 ±0.61 / 34.66 ms │ 30.76 / 32.16 ±0.92 / 33.30 ms │ no change │
│ QQuery 4 │ 269.05 / 271.35 ±2.25 / 275.47 ms │ 217.35 / 219.75 ±2.46 / 223.86 ms │ +1.23x faster │
│ QQuery 5 │ 274.65 / 287.34 ±15.38 / 311.07 ms │ 272.53 / 274.17 ±1.18 / 275.27 ms │ no change │
│ QQuery 6 │ 1.26 / 1.41 ±0.22 / 1.83 ms │ 1.30 / 1.44 ±0.22 / 1.88 ms │ no change │
│ QQuery 7 │ 13.13 / 13.21 ±0.08 / 13.34 ms │ 13.19 / 13.29 ±0.06 / 13.35 ms │ no change │
│ QQuery 8 │ 331.27 / 338.65 ±9.31 / 356.94 ms │ 326.08 / 329.64 ±2.10 / 331.74 ms │ no change │
│ QQuery 9 │ 465.04 / 469.20 ±2.92 / 473.48 ms │ 453.24 / 462.02 ±5.58 / 469.48 ms │ no change │
│ QQuery 10 │ 71.43 / 72.01 ±0.32 / 72.38 ms │ 69.37 / 70.17 ±0.41 / 70.46 ms │ no change │
│ QQuery 11 │ 81.47 / 82.42 ±1.08 / 84.46 ms │ 80.52 / 81.64 ±1.34 / 84.06 ms │ no change │
│ QQuery 12 │ 268.42 / 280.57 ±14.72 / 308.63 ms │ 265.57 / 272.39 ±5.14 / 277.48 ms │ no change │
│ QQuery 13 │ 414.77 / 426.12 ±9.20 / 436.40 ms │ 390.92 / 410.89 ±19.17 / 433.87 ms │ no change │
│ QQuery 14 │ 303.52 / 312.18 ±9.88 / 330.95 ms │ 322.59 / 334.42 ±12.14 / 350.28 ms │ 1.07x slower │
│ QQuery 15 │ 284.17 / 294.96 ±7.16 / 303.28 ms │ 326.14 / 334.37 ±7.81 / 347.55 ms │ 1.13x slower │
│ QQuery 16 │ 622.30 / 630.09 ±6.29 / 639.21 ms │ 718.09 / 725.89 ±5.24 / 732.43 ms │ 1.15x slower │
│ QQuery 17 │ 621.09 / 635.02 ±8.12 / 646.09 ms │ 634.22 / 665.61 ±27.09 / 712.72 ms │ no change │
│ QQuery 18 │ 1264.07 / 1337.93 ±50.85 / 1395.81 ms │ 1264.55 / 1375.45 ±89.29 / 1518.05 ms │ no change │
│ QQuery 19 │ 27.38 / 29.00 ±2.86 / 34.70 ms │ 28.74 / 28.88 ±0.15 / 29.16 ms │ no change │
│ QQuery 20 │ 517.30 / 523.70 ±6.22 / 535.49 ms │ 519.49 / 541.02 ±22.90 / 582.29 ms │ no change │
│ QQuery 21 │ 516.92 / 520.60 ±4.82 / 530.02 ms │ 523.74 / 527.77 ±2.52 / 531.35 ms │ no change │
│ QQuery 22 │ 998.75 / 1010.62 ±10.16 / 1027.44 ms │ 982.57 / 995.34 ±7.20 / 1002.06 ms │ no change │
│ QQuery 23 │ 3087.59 / 3138.92 ±60.87 / 3258.43 ms │ 3055.93 / 3129.85 ±47.21 / 3193.10 ms │ no change │
│ QQuery 24 │ 40.94 / 41.71 ±0.72 / 42.88 ms │ 40.97 / 44.76 ±3.95 / 49.74 ms │ 1.07x slower │
│ QQuery 25 │ 110.27 / 112.54 ±2.53 / 117.47 ms │ 109.63 / 111.71 ±2.05 / 115.21 ms │ no change │
│ QQuery 26 │ 41.50 / 46.59 ±7.96 / 62.39 ms │ 43.89 / 52.30 ±14.97 / 82.16 ms │ 1.12x slower │
│ QQuery 27 │ 515.13 / 534.88 ±12.89 / 549.71 ms │ 517.80 / 536.79 ±9.94 / 546.15 ms │ no change │
│ QQuery 28 │ 2981.23 / 3079.31 ±74.99 / 3204.79 ms │ 2895.42 / 2958.78 ±68.02 / 3088.00 ms │ no change │
│ QQuery 29 │ 41.29 / 45.29 ±7.63 / 60.53 ms │ 43.79 / 51.56 ±8.88 / 63.16 ms │ 1.14x slower │
│ QQuery 30 │ 317.14 / 320.12 ±1.97 / 323.00 ms │ 360.25 / 366.11 ±5.95 / 376.85 ms │ 1.14x slower │
│ QQuery 31 │ 290.61 / 309.38 ±10.68 / 323.64 ms │ 315.30 / 323.58 ±6.29 / 334.24 ms │ no change │
│ QQuery 32 │ 960.95 / 989.90 ±26.38 / 1024.28 ms │ 972.71 / 1061.20 ±72.56 / 1166.66 ms │ 1.07x slower │
│ QQuery 33 │ 1480.69 / 1549.94 ±76.26 / 1687.55 ms │ 1503.10 / 1571.48 ±84.85 / 1733.10 ms │ no change │
│ QQuery 34 │ 1475.04 / 1503.30 ±20.77 / 1528.53 ms │ 1485.80 / 1526.49 ±34.85 / 1587.97 ms │ no change │
│ QQuery 35 │ 274.67 / 316.10 ±54.02 / 420.17 ms │ 294.33 / 310.87 ±14.90 / 337.69 ms │ no change │
│ QQuery 36 │ 72.73 / 79.38 ±4.91 / 86.83 ms │ 75.55 / 80.22 ±2.92 / 84.01 ms │ no change │
│ QQuery 37 │ 35.35 / 36.47 ±0.98 / 37.58 ms │ 38.55 / 39.79 ±1.60 / 42.91 ms │ 1.09x slower │
│ QQuery 38 │ 41.61 / 44.04 ±1.50 / 46.36 ms │ 41.16 / 46.27 ±3.91 / 52.86 ms │ 1.05x slower │
│ QQuery 39 │ 144.92 / 155.36 ±7.28 / 164.82 ms │ 133.85 / 152.90 ±10.67 / 163.55 ms │ no change │
│ QQuery 40 │ 14.51 / 14.80 ±0.25 / 15.21 ms │ 14.10 / 15.81 ±2.52 / 20.78 ms │ 1.07x slower │
│ QQuery 41 │ 14.63 / 15.21 ±0.36 / 15.69 ms │ 13.51 / 13.66 ±0.14 / 13.88 ms │ +1.11x faster │
│ QQuery 42 │ 14.90 / 18.77 ±5.62 / 29.47 ms │ 12.68 / 15.49 ±5.06 / 25.60 ms │ +1.21x faster │
└───────────┴───────────────────────────────────────┴───────────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD) │ 19977.58ms │
│ Total Time (blocked-emit) │ 20160.17ms │
│ Average Time (HEAD) │ 464.59ms │
│ Average Time (blocked-emit) │ 468.84ms │
│ Queries Faster │ 4 │
│ Queries Slower │ 11 │
│ Queries with No Change │ 28 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴────────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**clickbench_partitioned — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 12.1 GiB |
| Avg memory | 4.3 GiB |
| CPU user | 1020.3s |
| CPU sys | 75.1s |
| Peak spill | 0 B |
**clickbench_partitioned — branch**
| Metric | Value |
|--------|-------|
| Wall time | 105.0s |
| Peak memory | 11.6 GiB |
| Avg memory | 4.5 GiB |
| CPU user | 1023.9s |
| CPU sys | 77.0s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5664212258
@rluvaton,
How would you like me to start on #24928? Is it possible to pull out independent pieces that can be merged into main incrementally?
This PR is around 1k LoC, which feels like a reasonable size for a first step. If you have a similar split in mind and can pull out a smaller PR, I'd be happy to review it and move forward from there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5664521086
Are you available for a quick talk on discord so I can share my current finding with you and get your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5663858119
I think we should coordinate on that since I have very similar implementation but mine is more complete and also I already did a lot of benchmarks on it
+ some decisions I did like:
- emit all should not return single large vec since it:
1. force a copy
2. allow for misuse
3. you want to emit all but in blocks (when spilling for example)
- you should not add block_len on the existing aggregates since the implementation should be build around the blocked impl
- having the same trait force emit all users to return large array rather than blocks and no emit block function
and many more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654056995
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5654044398-2333-74dtk 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark clickbench_partitioned
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654056968
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5654044398-2335-lrgk4 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#discussion_r4005099581
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,425 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`BlockedVec`]: per-group state stored in fixed-size blocks.
+
+use std::marker::PhantomData;
+use std::mem::size_of;
+
+use datafusion_common::utils::split_vec_min_alloc;
+
+/// Number of groups per block, as a power of two: `1 << DEFAULT_BLOCK_SHIFT`.
+///
+/// Every table with at most this many groups is a single block, which is
+/// read and updated exactly like a flat `Vec`. Larger tables pay one extra
+/// address-resolution pass per batch (see [`AddressResolver`]).
+pub const DEFAULT_BLOCK_SHIFT: u32 = 20;
+
+/// Per-group values stored in blocks of `1 << SHIFT` elements: group `g`
+/// lives in block `g >> SHIFT` at offset `g & ((1 << SHIFT) - 1)`.
+///
+/// Blocked storage lets a hash aggregation grow, emit and free its state one
+/// block at a time instead of as one contiguous allocation. The cost that
+/// made earlier blocked designs slower is *how* the state is addressed in the
+/// update loop, not the blocks themselves: `blocks[g >> SHIFT][g & MASK]` is
+/// a dependent load, so the CPU cannot issue the next group's cache miss
+/// until the block pointer has resolved, and the flat `Vec` wins on
+/// memory-level parallelism alone. [`AddressResolver`] resolves the
+/// addresses of a chunk of rows in a separate, independent pass first, after
+/// which the update loop issues its misses exactly as the flat loop does.
+#[derive(Debug)]
+pub struct BlockedVec<T, const SHIFT: u32 = DEFAULT_BLOCK_SHIFT> {
Review Comment:
I've had implementation for this in my huge WIP pr:
- #24928
some of my concerns, you require block size to be power of 2 (while probably the case, I don't know if we want to depend on that)
and many more, I really think we should sync on that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5650001408
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25246?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `94.13408%` with `42 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.96%. Comparing base ([`f8cc678`](https://app.codecov.io/gh/apache/datafusion/commit/f8cc678272c50c781a56b74b49912b999aebd89e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c3b786b`](https://app.codecov.io/gh/apache/datafusion/commit/c3b786b4cebe55fdbf754f12d97ca6642fecd7b6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 8 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25246?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...gregates/group\_values/single\_group\_by/primitive.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fsingle_group_by%2Fprimitive.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9zaW5nbGVfZ3JvdXBfYnkvcHJpbWl0aXZlLnJz) | 90.38% | [3 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...plan/src/aggregates/aggregate\_hash\_table/common.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Faggregate_hash_table%2Fcommon.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2FnZ3JlZ2F0ZV9oYXNoX3RhYmxlL2NvbW1vbi5ycw==) | 88.31% | [2 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...fusion/physical-plan/src/aggregates/hash\_stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fhash_stream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2hhc2hfc3RyZWFtLnJz) | 86.36% | [1 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...-common/src/aggregate/groups\_accumulator/blocks.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Ffunctions-aggregate-common%2Fsrc%2Faggregate%2Fgroups_accumulator%2Fblocks.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMtYWdncmVnYXRlLWNvbW1vbi9zcmMvYWdncmVnYXRlL2dyb3Vwc19hY2N1bXVsYXRvci9ibG9ja3MucnM=) | 96.94% | [5 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...common/src/aggregate/groups\_accumulator/prim\_op.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Ffunctions-aggregate-common%2Fsrc%2Faggregate%2Fgroups_accumulator%2Fprim_op.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMtYWdncmVnYXRlLWNvbW1vbi9zcmMvYWdncmVnYXRlL2dyb3Vwc19hY2N1bXVsYXRvci9wcmltX29wLnJz) | 94.11% | [0 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/physical-plan/src/aggregates/group\_values/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9tb2QucnM=) | 50.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25246?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25246 +/- ##
==========================================
+ Coverage 81.93% 81.96% +0.03%
==========================================
Files 1133 1134 +1
Lines 423529 424484 +955
Branches 423529 424484 +955
==========================================
+ Hits 347028 347949 +921
- Misses 55910 55915 +5
- Partials 20591 20620 +29
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25246?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654057037
🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c5654044398-2334-6r8xk 6.12.94+ #1 SMP Tue Aug 4 08:44:15 UTC 2026 aarch64 GNU/Linux`
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpcds
```
</details>
Results will be posted here when complete
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5654152526
🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/25246#issuecomment-5654044398)
**Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
Comparing blocked-emit (c3b786b4cebe55fdbf754f12d97ca6642fecd7b6) to f8cc678 (merge-base) [diff](https://github.com/apache/datafusion/compare/f8cc678272c50c781a56b74b49912b999aebd89e..c3b786b4cebe55fdbf754f12d97ca6642fecd7b6)
<details><summary>Run configuration</summary>
```yaml
run benchmark tpch
```
</details>
<details><summary>CPU Details (lscpu)</summary>
```
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: ARM
Model name: Neoverse-V2
Model: 1
Thread(s) per core: 1
Core(s) per cluster: 16
Socket(s): -
Cluster(s): 1
Stepping: r0p1
BogoMIPS: 2000.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache: 1 MiB (16 instances)
L1i cache: 1 MiB (16 instances)
L2 cache: 32 MiB (16 instances)
L3 cache: 80 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Mitigation; CSV2, BHB
Vulnerability Srbds: Not affected
Vulnerability Tsa: Not affected
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Not affected
```
</details>
<details><summary>Details</summary>
<p>
```
Comparing HEAD and blocked-emit
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 38.54 ms │ 38.40 ms │ no change │
│ QQuery 2 │ 18.61 ms │ 19.15 ms │ no change │
│ QQuery 3 │ 28.60 ms │ 28.88 ms │ no change │
│ QQuery 4 │ 17.49 ms │ 17.64 ms │ no change │
│ QQuery 5 │ 36.19 ms │ 36.12 ms │ no change │
│ QQuery 6 │ 16.78 ms │ 16.14 ms │ no change │
│ QQuery 7 │ 42.08 ms │ 41.01 ms │ no change │
│ QQuery 8 │ 40.86 ms │ 41.03 ms │ no change │
│ QQuery 9 │ 49.32 ms │ 49.59 ms │ no change │
│ QQuery 10 │ 43.09 ms │ 43.01 ms │ no change │
│ QQuery 11 │ 13.82 ms │ 14.10 ms │ no change │
│ QQuery 12 │ 24.23 ms │ 24.42 ms │ no change │
│ QQuery 13 │ 40.74 ms │ 40.28 ms │ no change │
│ QQuery 14 │ 24.92 ms │ 25.12 ms │ no change │
│ QQuery 15 │ 31.36 ms │ 32.08 ms │ no change │
│ QQuery 16 │ 13.69 ms │ 13.89 ms │ no change │
│ QQuery 17 │ 73.86 ms │ 68.09 ms │ +1.08x faster │
│ QQuery 18 │ 61.48 ms │ 61.81 ms │ no change │
│ QQuery 19 │ 33.24 ms │ 32.60 ms │ no change │
│ QQuery 20 │ 31.95 ms │ 31.58 ms │ no change │
│ QQuery 21 │ 57.08 ms │ 55.82 ms │ no change │
│ QQuery 22 │ 14.69 ms │ 14.68 ms │ no change │
└───────────┴──────────┴──────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 752.62ms │
│ Total Time (blocked-emit) │ 745.44ms │
│ Average Time (HEAD) │ 34.21ms │
│ Average Time (blocked-emit) │ 33.88ms │
│ Queries Faster │ 1 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 21 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴──────────┘
Distribution per query (min / mean ±stddev / max):
Comparing HEAD and blocked-emit
--------------------
Benchmark tpch_sf1.json
--------------------
┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query ┃ HEAD ┃ blocked-emit ┃ Change ┃
┡━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1 │ 38.54 / 40.39 ±2.14 / 44.23 ms │ 38.40 / 39.56 ±1.24 / 41.23 ms │ no change │
│ QQuery 2 │ 18.61 / 19.69 ±1.01 / 20.98 ms │ 19.15 / 19.56 ±0.28 / 19.93 ms │ no change │
│ QQuery 3 │ 28.60 / 29.33 ±1.17 / 31.66 ms │ 28.88 / 29.69 ±0.48 / 30.31 ms │ no change │
│ QQuery 4 │ 17.49 / 18.31 ±1.31 / 20.91 ms │ 17.64 / 17.85 ±0.24 / 18.31 ms │ no change │
│ QQuery 5 │ 36.19 / 37.06 ±1.14 / 39.32 ms │ 36.12 / 36.30 ±0.28 / 36.87 ms │ no change │
│ QQuery 6 │ 16.78 / 18.29 ±1.14 / 19.98 ms │ 16.14 / 16.32 ±0.15 / 16.58 ms │ +1.12x faster │
│ QQuery 7 │ 42.08 / 45.02 ±2.72 / 49.61 ms │ 41.01 / 42.85 ±1.34 / 44.99 ms │ no change │
│ QQuery 8 │ 40.86 / 41.91 ±1.36 / 44.56 ms │ 41.03 / 41.43 ±0.25 / 41.74 ms │ no change │
│ QQuery 9 │ 49.32 / 51.25 ±1.61 / 53.85 ms │ 49.59 / 51.13 ±1.06 / 52.55 ms │ no change │
│ QQuery 10 │ 43.09 / 44.06 ±1.05 / 45.43 ms │ 43.01 / 43.72 ±0.66 / 44.57 ms │ no change │
│ QQuery 11 │ 13.82 / 14.13 ±0.50 / 15.12 ms │ 14.10 / 14.60 ±0.66 / 15.89 ms │ no change │
│ QQuery 12 │ 24.23 / 24.51 ±0.30 / 25.05 ms │ 24.42 / 24.75 ±0.26 / 25.08 ms │ no change │
│ QQuery 13 │ 40.74 / 41.47 ±1.14 / 43.73 ms │ 40.28 / 41.35 ±1.65 / 44.58 ms │ no change │
│ QQuery 14 │ 24.92 / 25.02 ±0.07 / 25.14 ms │ 25.12 / 25.22 ±0.11 / 25.41 ms │ no change │
│ QQuery 15 │ 31.36 / 31.68 ±0.42 / 32.45 ms │ 32.08 / 32.57 ±0.56 / 33.60 ms │ no change │
│ QQuery 16 │ 13.69 / 14.20 ±0.41 / 14.87 ms │ 13.89 / 14.29 ±0.40 / 15.04 ms │ no change │
│ QQuery 17 │ 73.86 / 75.84 ±1.23 / 77.32 ms │ 68.09 / 69.70 ±1.71 / 72.55 ms │ +1.09x faster │
│ QQuery 18 │ 61.48 / 62.87 ±0.94 / 63.95 ms │ 61.81 / 62.62 ±0.88 / 64.12 ms │ no change │
│ QQuery 19 │ 33.24 / 34.01 ±0.90 / 35.76 ms │ 32.60 / 33.95 ±1.46 / 35.77 ms │ no change │
│ QQuery 20 │ 31.95 / 32.58 ±0.64 / 33.72 ms │ 31.58 / 32.11 ±0.67 / 33.42 ms │ no change │
│ QQuery 21 │ 57.08 / 57.83 ±0.65 / 58.88 ms │ 55.82 / 56.38 ±0.51 / 57.20 ms │ no change │
│ QQuery 22 │ 14.69 / 15.44 ±1.02 / 17.43 ms │ 14.68 / 15.04 ±0.42 / 15.85 ms │ no change │
└───────────┴────────────────────────────────┴────────────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Benchmark Summary ┃ ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Total Time (HEAD) │ 774.90ms │
│ Total Time (blocked-emit) │ 761.01ms │
│ Average Time (HEAD) │ 35.22ms │
│ Average Time (blocked-emit) │ 34.59ms │
│ Queries Faster │ 2 │
│ Queries Slower │ 0 │
│ Queries with No Change │ 20 │
│ Queries with Failure │ 0 │
└─────────────────────────────┴──────────┘
```
</p>
</details>
<details><summary>Resource Usage</summary>
**tpch — base (merge-base)**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.2 GiB |
| Avg memory | 508.3 MiB |
| CPU user | 21.7s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
**tpch — branch**
| Metric | Value |
|--------|-------|
| Wall time | 5.0s |
| Peak memory | 1.3 GiB |
| Avg memory | 512.5 MiB |
| CPU user | 21.1s |
| CPU sys | 1.9s |
| Peak spill | 0 B |
</details>
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on code in PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#discussion_r4005115153
##########
datafusion/functions-aggregate-common/src/aggregate/groups_accumulator/blocks.rs:
##########
@@ -0,0 +1,425 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`BlockedVec`]: per-group state stored in fixed-size blocks.
+
+use std::marker::PhantomData;
+use std::mem::size_of;
+
+use datafusion_common::utils::split_vec_min_alloc;
+
+/// Number of groups per block, as a power of two: `1 << DEFAULT_BLOCK_SHIFT`.
+///
+/// Every table with at most this many groups is a single block, which is
+/// read and updated exactly like a flat `Vec`. Larger tables pay one extra
+/// address-resolution pass per batch (see [`AddressResolver`]).
+pub const DEFAULT_BLOCK_SHIFT: u32 = 20;
+
+/// Per-group values stored in blocks of `1 << SHIFT` elements: group `g`
+/// lives in block `g >> SHIFT` at offset `g & ((1 << SHIFT) - 1)`.
+///
+/// Blocked storage lets a hash aggregation grow, emit and free its state one
+/// block at a time instead of as one contiguous allocation. The cost that
+/// made earlier blocked designs slower is *how* the state is addressed in the
+/// update loop, not the blocks themselves: `blocks[g >> SHIFT][g & MASK]` is
+/// a dependent load, so the CPU cannot issue the next group's cache miss
+/// until the block pointer has resolved, and the flat `Vec` wins on
+/// memory-level parallelism alone. [`AddressResolver`] resolves the
+/// addresses of a chunk of rows in a separate, independent pass first, after
+/// which the update loop issues its misses exactly as the flat loop does.
Review Comment:
I also implemented mmap vec that with it enabled we avoid the address computation and some more which results in better performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5664483183
The reason I did not split it already so we can discuss it.
I created a huge PR so:
1. We can see what every decision will take us and avoid iterating on main building blocks and creating a lot of breaking changes
2. See if this approach performance is even reasonable
3. See if the traits are allowing to implementing existing and company specific very complex aggregation.
I'm writing a summary of all my finding and will post it in the issue and I want to discuss it on the weekly meeting.
I don't want to merge prs yet that decide on the blocked path without us having it figured out and understanding the implications.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "adriangbot (via GitHub)" <gi...@apache.org>.
adriangbot commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5649976885
Hi @jayzhan211, thanks for the request (https://github.com/apache/datafusion/pull/25246#issuecomment-5649976775). Only whitelisted users can trigger benchmarks. Allowed users: [2010YOUY01](https://github.com/2010YOUY01), [Dandandan](https://github.com/Dandandan), [Fokko](https://github.com/Fokko), [Jefffrey](https://github.com/Jefffrey), [Omega359](https://github.com/Omega359), [Rachelint](https://github.com/Rachelint), [Rich-T-kid](https://github.com/Rich-T-kid), [adriangb](https://github.com/adriangb), [alamb](https://github.com/alamb), [asubiotto](https://github.com/asubiotto), [avantgardnerio](https://github.com/avantgardnerio), [brunal](https://github.com/brunal), [buraksenn](https://github.com/buraksenn), [cetra3](https://github.com/cetra3), [codephage2020](https://github.com/codephage2020), [coderfender](https://github.com/coderfender), [comphead](https://github.com/comphead), [erenavsarogullari](https://github.com/erenavsarogullari), [etseidl](https://github.com/etseidl),
[friendlymatthew](https://github.com/friendlymatthew), [gabotechs](https://github.com/gabotechs), [geoffreyclaude](https://github.com/geoffreyclaude), [grtlr](https://github.com/grtlr), [haohuaijin](https://github.com/haohuaijin), [jonathanc-n](https://github.com/jonathanc-n), [kevinjqliu](https://github.com/kevinjqliu), [klion26](https://github.com/klion26), [kosiew](https://github.com/kosiew), [kumarUjjawal](https://github.com/kumarUjjawal), [kunalsinghdadhwal](https://github.com/kunalsinghdadhwal), [liamzwbao](https://github.com/liamzwbao), [mbutrovich](https://github.com/mbutrovich), [mkleen](https://github.com/mkleen), [mzabaluev](https://github.com/mzabaluev), [neilconway](https://github.com/neilconway), [rluvaton](https://github.com/rluvaton), [sdf-jkl](https://github.com/sdf-jkl), [timsaucer](https://github.com/timsaucer), [xudong963](https://github.com/xudong963), [zhuqi-lucas](https://github.com/zhuqi-lucas).
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: blocked hash aggregation state via an independent address-resolve pass, with block-wise output release [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #25246:
URL: https://github.com/apache/datafusion/pull/25246#issuecomment-5649976775
run benchmarks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org