You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bvolpato (via GitHub)" <gi...@apache.org> on 2026/05/31 07:25:40 UTC

[PR] fix(substrait): preserve window function semantics [datafusion]

bvolpato opened a new pull request, #22664:
URL: https://github.com/apache/datafusion/pull/22664

   ## Which issue does this PR close?
   
   - Closes #22661.
   
   ## Rationale for this change
   
   Substrait logical-plan conversion can currently return a valid window plan with
   different semantics from its input. On current `main` (`496f2c2065c0`), a
   round trip of:
   
   ```sql
   SELECT array_agg(val) IGNORE NULLS OVER (
     ORDER BY ts ROWS BETWEEN 1 PRECEDING AND CURRENT ROW
   )
   ```
   
   changes expected output `[A], [A], [C], [C], [E]` to
   `[A], [A, NULL], [NULL, C], [C, NULL], [NULL, E]`.
   
   Likewise, `window.slt:909` uses finite interval `RANGE` bounds; round trip
   changes bounded counts such as `6, 2, 1` into partition-wide counts `8, 8, 8`.
   Returning changed results is worse than reporting unsupported conversion.
   
   ## What changes are included in this PR?
   
   - Encode and decode window `DISTINCT` through Substrait
     `AggregationInvocation`.
   - Reject explicit window null treatment and `FILTER`, which the current
     Substrait window expression path does not encode.
   - Reject finite frame offsets which cannot be represented as Substrait integer
     offsets instead of widening them to `UNBOUNDED`.
   - Add logical round-trip coverage for `DISTINCT` and fail-closed coverage for
     `IGNORE NULLS`, `RESPECT NULLS`, `FILTER`, and interval `RANGE` frames.
   
   ## Are these changes tested?
   
   - `cargo fmt --all`
   - `cargo clippy --all-targets --all-features -- -D warnings`
   - `cargo test -p datafusion-substrait`
   
   Pre-fix reproduction:
   
   ```bash
   cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip array_agg_sliding_window.slt:150 window.slt:909
   ```
   
   - `array_agg_sliding_window.slt:150` produced null-containing arrays despite
     `IGNORE NULLS`.
   - `window.slt:909` produced partition-wide counts for finite interval bounds.
   
   Post-fix targeted checks:
   
   ```bash
   cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip array_agg_sliding_window.slt:150
   cargo test -p datafusion-sqllogictest --test sqllogictests --features substrait -- --substrait-round-trip window.slt:909
   ```
   
   - The first now fails conversion with `Window functions with IGNORE NULLS are
     not supported in Substrait`.
   - The second now fails conversion with `Unsupported Substrait window frame
     offset: 1 DAY`.
   - Selecting `window.slt:909` also runs unrelated existing `EXPLAIN` records in
     that fixture, which report separate plan-output mismatches after the target
     conversion error.
   
   ## Are there any user-facing changes?
   
   Substrait producers now return explicit unsupported-feature errors for window
   null treatment, window `FILTER`, and finite non-integer window-frame offsets
   instead of producing plans with altered semantics. Window `DISTINCT` now
   round-trips correctly.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(substrait): preserve window function semantics [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5474510850

   Sure! taking a llok


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(substrait): preserve window function semantics [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5080652047

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `72.34043%` with `13 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 80.65%. Comparing base ([`36c417b`](https://app.codecov.io/gh/apache/datafusion/commit/36c417b6c14244526c81ee5dc4210740ab7b44f8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4a2a1e7`](https://app.codecov.io/gh/apache/datafusion/commit/4a2a1e72449f4f005494b39e65dfc8351f4818fe?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../src/logical\_plan/producer/expr/window\_function.rs](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&filepath=datafusion%2Fsubstrait%2Fsrc%2Flogical_plan%2Fproducer%2Fexpr%2Fwindow_function.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zdWJzdHJhaXQvc3JjL2xvZ2ljYWxfcGxhbi9wcm9kdWNlci9leHByL3dpbmRvd19mdW5jdGlvbi5ycw==) | 75.00% | [8 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [.../src/logical\_plan/consumer/expr/window\_function.rs](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&filepath=datafusion%2Fsubstrait%2Fsrc%2Flogical_plan%2Fconsumer%2Fexpr%2Fwindow_function.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zdWJzdHJhaXQvc3JjL2xvZ2ljYWxfcGxhbi9jb25zdW1lci9leHByL3dpbmRvd19mdW5jdGlvbi5ycw==) | 57.14% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/22664?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #22664      +/-   ##
   ==========================================
   - Coverage   80.65%   80.65%   -0.01%     
   ==========================================
     Files        1092     1092              
     Lines      371140   371150      +10     
     Branches   371140   371150      +10     
   ==========================================
   + Hits       299353   299354       +1     
   - Misses      53933    53941       +8     
   - Partials    17854    17855       +1     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/22664?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(substrait): preserve window function semantics [datafusion]

Posted by "bvolpato (via GitHub)" <gi...@apache.org>.
bvolpato commented on PR #22664:
URL: https://github.com/apache/datafusion/pull/22664#issuecomment-5464290835

   @gabotechs, would you have some time to review this one?
   
   it fixes two cases where Substrait round-trips silently change window semantics (`IGNORE NULLS` and interval `RANGE` frames). unsupported cases now fail explicitly instead of returning different results, and `DISTINCT` round-trips correctly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix(substrait): preserve window function semantics [datafusion]

Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs merged PR #22664:
URL: https://github.com/apache/datafusion/pull/22664


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org