You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "avantgardnerio (via GitHub)" <gi...@apache.org> on 2026/07/30 16:34:29 UTC
[PR] feat(physical-plan): expose finalized Accumulator state on BoundedWindowAggExec [datafusion]
avantgardnerio opened a new pull request, #24007:
URL: https://github.com/apache/datafusion/pull/24007
## Summary
Adds a getter on `BoundedWindowAggExec` that returns the finalized `Accumulator::state()` for each aggregate window expression at partition close, keyed by PARTITION BY tuple. This lets callers observe per-partition final aggregate state without emitting it as an output column on every row.
The change is purely additive:
- **New public type alias** `FinalizedPartitionState = HashMap<PartitionKey, Vec<Option<Vec<ScalarValue>>>>` — one `Accumulator::state()` per (PARTITION BY tuple, window expression) pair. Non-aggregate window functions (`row_number`, `rank`, `lead`/`lag`, ...) occupy `None` at their slot in the inner `Vec`.
- **New field** `finalized_state: Arc<[Mutex<FinalizedPartitionState>]>` on `BoundedWindowAggExec` — one slot per output partition.
- **New public method** `BoundedWindowAggExec::finalized_partition_state(partition)` — returns the current snapshot for a given output partition.
- **Write site** at the top of `prune_state` in `BoundedWindowAggStream` — snapshots every entry whose `state.is_end` is set, immediately before either `prune_out_columns` or `prune_partition_batches` drops it. This covers both mid-stream partition close (as SQL PARTITION BY groups end) and end-of-stream with a single write path.
No trait changes: the existing public `Accumulator::state()` method (already used by the group-by Partial/Final protocol) is the state source. Behavior without a reader is unchanged — the slots simply hold state that no one reads.
## Motivation
Distributed executors that split a windowed aggregate across multiple tasks need each task's final per-partition state to compose results across tasks (for aggregates like `AVG` whose state is not recoverable from the output column, or for sketch-backed aggregates whose state is a merge-able structure). Exposing state at partition-close time keeps the observation off the output stream — no state emitted as row-level columns.
## Scope
`BoundedWindowAggExec` only. `WindowAggExec` doesn't hold `Accumulator` boxes across its partition loop — its accumulators are constructed and discarded inside `WindowExpr::evaluate` — so the same shape there would need either a `WindowExpr` trait extension or a compute-side restructure. Deferring that until a caller actually needs it; distributed callers can canonicalize to `BoundedWindowAggExec` (with `InputOrderMode::Sorted`) when the input is already ordered, which is the shape-fitting choice regardless.
## Test plan
- [x] New unit test `finalized_partition_state_captures_sum_per_partition_key`: runs cumulative `SUM(v) OVER (PARTITION BY pk)` on two partition keys, drains the stream, verifies each key's finalized state contains the expected sum. Also exercises the out-of-range partition error path.
- [x] `cargo test -p datafusion-physical-plan --lib` — 1610 tests pass, no regressions.
- [x] `cargo clippy -p datafusion-physical-plan --all-targets` — clean.
- [x] `cargo check --workspace` — clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): expose finalized Accumulator state on BoundedWindowAggExec [datafusion]
Posted by "avantgardnerio (via GitHub)" <gi...@apache.org>.
avantgardnerio closed pull request #24007: feat(physical-plan): expose finalized Accumulator state on BoundedWindowAggExec
URL: https://github.com/apache/datafusion/pull/24007
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat(physical-plan): expose finalized Accumulator state on BoundedWindowAggExec [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24007:
URL: https://github.com/apache/datafusion/pull/24007#issuecomment-5134066325
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24007?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `79.48718%` with `24 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.84%. Comparing base ([`62cfc0c`](https://app.codecov.io/gh/apache/datafusion/commit/62cfc0ceae30f92e12a34ee79d2a1be8c03b0d80?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`65ac32e`](https://app.codecov.io/gh/apache/datafusion/commit/65ac32e925ebf4d9d3ef4389f48eb67d03b23827?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 3 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24007?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ysical-plan/src/windows/bounded\_window\_agg\_exec.rs](https://app.codecov.io/gh/apache/datafusion/pull/24007?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fwindows%2Fbounded_window_agg_exec.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy93aW5kb3dzL2JvdW5kZWRfd2luZG93X2FnZ19leGVjLnJz) | 79.48% | [10 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24007?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24007 +/- ##
==========================================
- Coverage 80.85% 80.84% -0.01%
==========================================
Files 1096 1096
Lines 373911 374053 +142
Branches 373911 374053 +142
==========================================
+ Hits 302329 302418 +89
- Misses 53547 53574 +27
- Partials 18035 18061 +26
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24007?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org