You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/06/01 19:19:46 UTC
[I] Natively support time-window grouping expressions: window, session_window, window_time [datafusion-comet]
andygrove opened a new issue, #4553:
URL: https://github.com/apache/datafusion-comet/issues/4553
## Background
Spark's time-window grouping expressions currently fall back to Spark in Comet:
- `window(timeColumn, windowDuration, [slideDuration], [startTime])` (`TimeWindow`) - tumbling/sliding windows, common in batch aggregation (`GROUP BY window(ts, '1 hour')`), not just streaming.
- `session_window(timeColumn, gapDuration)` (`SessionWindow`) - session windows.
- `window_time(window)` (`WindowTime`) - extracts the event time from a window column.
Comet has no serde for `TimeWindow` / `SessionWindow` / `WindowTime` today, so any query using them falls back.
## Notes
These are not plain scalar functions: Spark's analyzer (`TimeWindowing` / `SessionWindowing` rules) rewrites `window()` / `session_window()` into an `Expand` plus grouping on a computed window struct. Native support would need to handle the rewritten form (struct construction and the window-boundary arithmetic) and the grouping that follows.
`window()` over a fixed duration is the most commonly used of the three and would be the natural starting point.
## Acceptance criteria
- `window`, `session_window`, and `window_time` execute natively in Comet and match Spark.
- Add SQL file test coverage under `expressions/datetime/`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Natively support time-window grouping expressions: window, session_window, window_time [datafusion-comet]
Posted by "dwsmith1983 (via GitHub)" <gi...@apache.org>.
dwsmith1983 commented on issue #4553:
URL: https://github.com/apache/datafusion-comet/issues/4553#issuecomment-5637149401
Two of the three landed in #4784 (f994b2359): batch tumbling and sliding `window(...)` grouping and `window_time` now run natively, and docs/source/user-guide/latest/expressions.md lists both as supported. What remains is `session_window`, which needs a native `UpdatingSessionsExec`/`MergingSessionsExec` and is already tracked on its own in #4785, also referenced from the `session_window` row of the same doc. Since the remaining work has a dedicated issue, suggest closing this one in favor of #4785.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Natively support time-window grouping expressions: window, session_window, window_time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #4553: Natively support time-window grouping expressions: window, session_window, window_time
URL: https://github.com/apache/datafusion-comet/issues/4553
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Natively support time-window grouping expressions: window, session_window, window_time [datafusion-comet]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on issue #4553:
URL: https://github.com/apache/datafusion-comet/issues/4553#issuecomment-4747338480
@andygrove those are structured streaming functions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Natively support time-window grouping expressions: window, session_window, window_time [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #4553:
URL: https://github.com/apache/datafusion-comet/issues/4553#issuecomment-5861039598
#4784 made batch tumbling and sliding window() grouping and window_time run natively, and window.sql covers both. session_window needs native UpdatingSessionsExec and MergingSessionsExec, which #4785 tracks, so I'm closing this in favour of that issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org