You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bvolpato (via GitHub)" <gi...@apache.org> on 2026/05/31 07:25:40 UTC
[PR] fix(substrait): preserve window function semantics [datafusion]
bvolpato opened a new pull request, #22664:
URL: https://github.com/apache/datafusion/pull/22664
## Which issue does this PR close?
- Closes #22661.
## Rationale for this change
Substrait logical-plan conversion can currently return a valid window plan with
different semantics from its input. On current `main` (`496f2c2065c0`), a
round trip of:
```sql
SELECT array_agg(val) IGNORE NULLS OVER (
ORDER BY ts ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
)
```
changes expected output `[A], [A], [C], [C], [E]` to
`[A], [A, NULL], [NULL, C], [C, NULL], [NULL, E]`.
Likewise, `window.slt:909` uses finite interval `RANGE` bounds; round trip
changes bounded counts such as `6, 2, 1` into partition-wide counts `8, 8, 8`.
Returning changed results is worse than reporting unsupported conversion.
## What changes are included in this PR?
- Encode and decode window `DISTINCT` through Substrait
`AggregationInvocation`.
- Reject explicit window null treatment and `FILTER`, which the current
Substrait window expression path does not encode.
- Reject finite frame offsets which cannot be represented as Substrait integer
offsets instead of widening them to `UNBOUNDED`.
- Add logical round-trip coverage for `DISTINCT` and fail-closed coverage for
`IGNORE NULLS`, `RESPECT NULLS`, `FILTER`, and interval `RANGE` frames.
## Are these changes tested?
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion-substrait`
Pre-fix reproduction:
```bash
cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip array_agg_sliding_window.slt:150 window.slt:909
```
- `array_agg_sliding_window.slt:150` produced null-containing arrays despite
`IGNORE NULLS`.
- `window.slt:909` produced partition-wide counts for finite interval bounds.
Post-fix targeted checks:
```bash
cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip array_agg_sliding_window.slt:150
cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip window.slt:909
```
- The first now fails conversion with `Window functions with IGNORE NULLS are
not supported in Substrait`.
- The second now fails conversion with `Unsupported Substrait window frame
offset: 1 DAY`.
- Selecting `window.slt:909` also runs unrelated existing `EXPLAIN` records in
that fixture, which report separate plan-output mismatches after the target
conversion error.
## Are there any user-facing changes?
Substrait producers now return explicit unsupported-feature errors for window
null treatment, window `FILTER`, and finite non-integer window-frame offsets
instead of producing plans with altered semantics. Window `DISTINCT` now
round-trips correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(substrait): preserve window function semantics [datafusion]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5474510850
Sure! taking a llok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(substrait): preserve window function semantics [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5080652047
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `72.34043%` with `13 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.65%. Comparing base ([`36c417b`](https://app.codecov.io/gh/apache/datafusion/commit/36c417b6c14244526c81ee5dc4210740ab7b44f8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4a2a1e7`](https://app.codecov.io/gh/apache/datafusion/commit/4a2a1e72449f4f005494b39e65dfc8351f4818fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../src/logical\_plan/producer/expr/window\_function.rs](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&filepath=datafusion%2Fsubstrait%2Fsrc%2Flogical_plan%2Fproducer%2Fexpr%2Fwindow_function.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zdWJzdHJhaXQvc3JjL2xvZ2ljYWxfcGxhbi9wcm9kdWNlci9leHByL3dpbmRvd19mdW5jdGlvbi5ycw==) | 75.00% | [8 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../src/logical\_plan/consumer/expr/window\_function.rs](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&filepath=datafusion%2Fsubstrait%2Fsrc%2Flogical_plan%2Fconsumer%2Fexpr%2Fwindow_function.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zdWJzdHJhaXQvc3JjL2xvZ2ljYWxfcGxhbi9jb25zdW1lci9leHByL3dpbmRvd19mdW5jdGlvbi5ycw==) | 57.14% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #22664 +/- ##
==========================================
- Coverage 80.65% 80.65% -0.01%
==========================================
Files 1092 1092
Lines 371140 371150 +10
Branches 371140 371150 +10
==========================================
+ Hits 299353 299354 +1
- Misses 53933 53941 +8
- Partials 17854 17855 +1
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(substrait): preserve window function semantics [datafusion]
Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5464290835
@gabotechs, would you have some time to review this one?
it fixes two cases where Substrait round-trips silently change window semantics (`IGNORE NULLS` and interval `RANGE` frames). unsupported cases now fail explicitly instead of returning different results, and `DISTINCT` round-trips correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix(substrait): preserve window function semantics [datafusion]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs merged PR #22664:
URL: https://github.com/apache/datafusion/pull/22664
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org