You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/09/25 01:32:09 UTC

[PR] feat: bucket the hash table of single stage aggregations that run on every partition [datafusion]

jayzhan211 opened a new pull request, #25725:
URL: https://github.com/apache/datafusion/pull/25725

   ## Which issue does this PR close?
   
   - Part of #20773 and #24704. Does not close them.
   
   **Stacked on #25724 — please review only the last commit.** The first six commits are that PR; GitHub shows them here because a fork PR cannot be based on another fork branch. This one is held as a draft until #25724 merges, at which point it will rebase down to a single commit.
   
   ## Rationale for this change
   
   #25724 lets a *final* hash aggregation move a table that has outgrown the CPU caches into hash buckets. The same problem exists for a single-stage aggregation — `AggregateMode::SinglePartitioned`, which is what a `GROUP BY` over already-partitioned input plans to, typically an aggregation over another aggregation. Its table grows the same way and pays the same cache miss per probe.
   
   This PR reuses the bucket driver extracted in #25724 (`aggregates/bucketed_aggregation.rs`) from `SingleHashAggregateStream`.
   
   A lone `Single` stream — one partition, no repartition below it — is deliberately **not** bucketed. Measured with `target_partitions = 1` it is 6-25% slower, because every row is then aggregated twice (once into the capped raw table, once into its bucket) and a single uncontended thread suffers far less from a large table in the first place.
   
   Evidence so far, on a nested aggregation over TPC-H SF10 `lineitem` (`GROUP BY l_orderkey, l_linenumber, mn` over a `GROUP BY l_orderkey, l_linenumber`), 12 partitions, threshold 262144: **0.85x time and 0.46x peak RSS**. Those numbers come from the whole series measured together; isolated before/after numbers for this commit alone will be added before this leaves draft.
   
   ## What changes are included in this PR?
   
   `SingleHashAggregateStream` gains the same three steps the final stream has: split its table into buckets once it reaches the threshold, route further input, and hand the buckets to `BucketedAggregation::output_stream` at end of input. It is enabled only when `agg.mode == AggregateMode::SinglePartitioned`, and the existing exclusions (nested state, soft group limit, no spill manager) carry over unchanged.
   
   ## What is the testing strategy for this PR?
   
   - Unit tests in `aggregates/hash_stream.rs`: a single-stage bucketed aggregation returns the same rows as one table, it spills its buckets under a memory limit, and a lone `Single` aggregation keeps its table.
   - `aggregate_bucketed.slt` gains a nested-aggregation section that asserts `mode=SinglePartitioned` with `bucket_splits`, run with the option off and at 100 groups with identical results, at both 4 and 1 partitions.
   
   ## Are there any user-facing changes?
   
   None beyond #25724: the same experimental option now also applies to single-stage aggregations that run on every partition.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: bucket the hash table of single stage aggregations that run on every partition [datafusion]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #25725:
URL: https://github.com/apache/datafusion/pull/25725#issuecomment-5825528194

   <!-- semver-check-comment -->
   Thank you for opening this pull request!
   
   Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
   
   <details>
   <summary>Details</summary>
   
   ```
        Cloning apache/main
       Building datafusion-common v55.1.0 (current)
          Built [  34.052s] (current)
        Parsing datafusion-common v55.1.0 (current)
         Parsed [   0.064s] (current)
       Building datafusion-common v55.1.0 (baseline)
          Built [  33.768s] (baseline)
        Parsing datafusion-common v55.1.0 (baseline)
         Parsed [   0.064s] (baseline)
       Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.712s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
   
   --- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
   
   Description:
   A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
           ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
          impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/constructible_struct_adds_field.ron
   
   Failed in:
     field ExecutionOptions.hash_aggregate_bucket_threshold in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:896
   
        Summary semver requires new major version: 1 major and 0 minor checks failed
       Finished [  70.007s] datafusion-common
       Building datafusion-physical-plan v55.1.0 (current)
          Built [  38.920s] (current)
        Parsing datafusion-physical-plan v55.1.0 (current)
         Parsed [   0.171s] (current)
       Building datafusion-physical-plan v55.1.0 (baseline)
          Built [  39.142s] (baseline)
        Parsing datafusion-physical-plan v55.1.0 (baseline)
         Parsed [   0.170s] (baseline)
       Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.637s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [  81.457s] datafusion-physical-plan
       Building datafusion-sqllogictest v55.1.0 (current)
          Built [ 101.943s] (current)
        Parsing datafusion-sqllogictest v55.1.0 (current)
         Parsed [   0.021s] (current)
       Building datafusion-sqllogictest v55.1.0 (baseline)
          Built [ 101.996s] (baseline)
        Parsing datafusion-sqllogictest v55.1.0 (baseline)
         Parsed [   0.023s] (baseline)
       Checking datafusion-sqllogictest v55.1.0 -> v55.1.0 (no change; assume patch)
        Checked [   0.094s] 223 checks: 223 pass, 31 skip
        Summary no semver update required
       Finished [ 207.010s] datafusion-sqllogictest
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: bucket the hash table of single stage aggregations that run on every partition [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #25725:
URL: https://github.com/apache/datafusion/pull/25725#issuecomment-5825196939

   Stacked on #25724: the first six commits are that PR. Only the last commit, `feat: bucket the hash table of single stage aggregations that run on every partition`, is new here. Held as a draft until #25724 merges.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: bucket the hash table of single stage aggregations that run on every partition [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25725:
URL: https://github.com/apache/datafusion/pull/25725#issuecomment-5825716620

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25725?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `88.58965%` with `161 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.50%. Comparing base ([`fb42cf5`](https://app.codecov.io/gh/apache/datafusion/commit/fb42cf5911faf1c05f6ccd5bde3d9e2300d35b76?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`e6a8ac4`](https://app.codecov.io/gh/apache/datafusion/commit/e6a8ac409c95175dbf5fcb68425a054db7a7920f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 2 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25725?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [...fusion/physical-plan/src/aggregates/hash\_stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fhash_stream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2hhc2hfc3RyZWFtLnJz) | 89.34% | [10 Missing and 47 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...sion/physical-plan/src/aggregates/final\_buckets.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Ffinal_buckets.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2ZpbmFsX2J1Y2tldHMucnM=) | 86.69% | [8 Missing and 29 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ysical-plan/src/aggregates/bucketed\_aggregation.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fbucketed_aggregation.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2J1Y2tldGVkX2FnZ3JlZ2F0aW9uLnJz) | 85.06% | [10 Missing and 23 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...sion/physical-plan/src/aggregates/single\_stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fsingle_stream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL3NpbmdsZV9zdHJlYW0ucnM=) | 80.37% | [16 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...src/aggregates/aggregate\_hash\_table/final\_table.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Faggregate_hash_table%2Ffinal_table.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2FnZ3JlZ2F0ZV9oYXNoX3RhYmxlL2ZpbmFsX3RhYmxlLnJz) | 90.62% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...gregates/group\_values/multi\_group\_by/bytes\_view.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fmulti_group_by%2Fbytes_view.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9tdWx0aV9ncm91cF9ieS9ieXRlc192aWV3LnJz) | 97.34% | [1 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../src/aggregates/group\_values/multi\_group\_by/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fmulti_group_by%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9tdWx0aV9ncm91cF9ieS9tb2QucnM=) | 92.30% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...plan/src/aggregates/aggregate\_hash\_table/common.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Faggregate_hash_table%2Fcommon.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2FnZ3JlZ2F0ZV9oYXNoX3RhYmxlL2NvbW1vbi5ycw==) | 96.96% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...n/physical-plan/src/aggregates/group\_values/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Faggregates%2Fgroup_values%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9hZ2dyZWdhdGVzL2dyb3VwX3ZhbHVlcy9tb2QucnM=) | 86.66% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25725?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #25725      +/-   ##
   ==========================================
   + Coverage   82.49%   82.50%   +0.01%     
   ==========================================
     Files        1140     1142       +2     
     Lines      439034   439875     +841     
     Branches   439034   439875     +841     
   ==========================================
   + Hits       362165   362916     +751     
   - Misses      54962    55005      +43     
   - Partials    21907    21954      +47     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25725?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org