You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "benbellick (via GitHub)" <gi...@apache.org> on 2026/10/17 16:19:27 UTC
[PR] feat: reuse range partitioning for date_trunc aggregation [datafusion]
benbellick opened a new pull request, #26449:
URL: https://github.com/apache/datafusion/pull/26449
## Which issue does this PR close?
Closes apache/datafusion#26448.
Related to apache/datafusion#23569, apache/datafusion#25344, and apache/datafusion#25345.
## Rationale for this change
This PR implements a focused subset of the optimization in [Nga's PR #24501](https://github.com/apache/datafusion/pull/24501) for a known-safe case. The plan is to iteratively expand support to more safe cases, following [Gene's review feedback](https://github.com/apache/datafusion/pull/24501#issuecomment-5589790676).
Suppose `events` is range-partitioned on `timestamp`, with partitions split at hourly boundaries.
```sql
SELECT
key,
date_trunc('hour', timestamp) AS hour,
SUM(value)
FROM events
GROUP BY key, date_trunc('hour', timestamp);
```
Each `(key, hour)` group belongs entirely to one input partition. The simplified physical plans below omit projections and use `hour` for the grouping expression.
**Without the optimization**
```text
AggregateExec: mode=FinalPartitioned, gby=[key, hour], aggr=[sum(value)]
RepartitionExec: Hash([key, hour])
AggregateExec: mode=Partial, gby=[key, hour], aggr=[sum(value)]
Scan: Range(timestamp), hourly splits
```
**With the optimization**
```text
AggregateExec: mode=SinglePartitioned, gby=[key, hour], aggr=[sum(value)]
Scan: Range(timestamp), hourly splits
```
The new locality check is specific to aggregates. General distribution satisfaction and join-layout checks are unchanged.
## What changes are included in this PR?
Add a default-deny capability that identifies functions and argument types safe for range-key analysis. Check expression ordering and arguments, then evaluate both sides of each split to prove groups stay within partitions.
The optimizer calls it through this path.
```mermaid
flowchart TD
A["EnsureRequirements checks aggregate input"] --> B["AggregateExec::input_keeps_groups_local"]
B --> C["Partitioning::keeps_keys_local"]
C --> D{"Function admitted, expression supported, and split boundaries safe?"}
D -->|Yes| E["Can skip the aggregate hash shuffle"]
D -->|No or unknown| F["Retain normal repartitioning"]
```
**Support is intentionally limited to timezone-less `date_trunc` over single-key timestamp ranges, with literal non-range arguments.** `date_bin` remains excluded pending its safety audit, including null behavior. Compound range keys remain follow-up work in apache/datafusion#24644.
Keep the shuffle when buckets cross partitions or the checks fail. Grouping sets still use the existing distribution checks.
## What is the testing strategy for this PR?
Added multiple unit and integration tests.
I'm happy to add a benchmark if maintainers would find it useful.
## Are there any user-facing changes?
Eligible `date_trunc` aggregations can skip a hash shuffle. Results are unchanged.
---
Note: This PR was developed with AI assistance. All changes have been reviewed, and I take full responsibility for this contribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: reuse range partitioning for date_trunc aggregation [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #26449:
URL: https://github.com/apache/datafusion/pull/26449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: reuse range partitioning for date_trunc aggregation [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26449:
URL: https://github.com/apache/datafusion/pull/26449#issuecomment-6237119851
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26449?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `95.79618%` with `33 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.92%. Comparing base ([`484183d`](https://app.codecov.io/gh/apache/datafusion/commit/484183dc34ea565190d9cd4a66db6d24789f95d0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a6482e5`](https://app.codecov.io/gh/apache/datafusion/commit/a6482e56a4cb9ef06b4efdeeecdd37b64f67a21e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26449?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-expr/src/partitioning.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fpartitioning.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9wYXJ0aXRpb25pbmcucnM=) | 95.84% | [9 Missing and 16 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/udf.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Fudf.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy91ZGYucnM=) | 77.77% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/functions/src/datetime/date\_trunc.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Ffunctions%2Fsrc%2Fdatetime%2Fdate_trunc.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9mdW5jdGlvbnMvc3JjL2RhdGV0aW1lL2RhdGVfdHJ1bmMucnM=) | 98.59% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...on/physical-expr/src/equivalence/properties/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fequivalence%2Fproperties%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9lcXVpdmFsZW5jZS9wcm9wZXJ0aWVzL21vZC5ycw==) | 94.11% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/physical-expr/src/scalar\_function.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Fphysical-expr%2Fsrc%2Fscalar_function.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1leHByL3NyYy9zY2FsYXJfZnVuY3Rpb24ucnM=) | 92.30% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...atafusion/physical-optimizer/src/sanity\_checker.rs](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&filepath=datafusion%2Fphysical-optimizer%2Fsrc%2Fsanity_checker.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1vcHRpbWl6ZXIvc3JjL3Nhbml0eV9jaGVja2VyLnJz) | 90.90% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26449?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26449 +/- ##
==========================================
+ Coverage 82.89% 82.92% +0.02%
==========================================
Files 1154 1154
Lines 457630 458406 +776
Branches 457630 458406 +776
==========================================
+ Hits 379370 380114 +744
- Misses 55256 55267 +11
- Partials 23004 23025 +21
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26449?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org