You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yadavay-amzn (via GitHub)" <gi...@apache.org> on 2026/07/02 00:47:44 UTC
[PR] [SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps [spark]
yadavay-amzn opened a new pull request, #56951:
URL: https://github.com/apache/spark/pull/56951
### What changes were proposed in this pull request?
Extends `window`, `session_window`, and `window_time` to accept nanosecond-precision timestamp columns (`TimestampNTZNanosType`, `TimestampLTZNanosType`). Window boundaries are computed on the nanosecond scale: durations are scaled x1000 with `Math.multiplyExact`, timestamp <-> epoch-nanos round-trips go through `PreciseTimestampNanosConversion`, and `window_time` subtracts 1 nanosecond (vs 1 microsecond for micros input). The window start/end preserve the input's nanosecond precision.
This also updates the batch `session_window` aggregation iterators (`MergingSessionsIterator` / `UpdatingSessionsIterator`) to read and compare session boundaries via the nanosecond-aware `TimestampNanosVal` accessors through a shared `SessionNanosHelper`. These sort-merge iterators are on the shared (non-streaming) batch aggregation path, and `getLong` would misread the variable-length `TimestampNanosVal` representation - so the feature cannot function correctly without them.
### Why are the changes needed?
Part of nanosecond-precision timestamp support (SPARK-56822), building on the event-time watermark support in SPARK-57830. Previously `window` / `session_window` / `window_time` rejected nanosecond-precision timestamp columns.
### Does this PR introduce _any_ user-facing change?
Yes - `window`, `session_window`, and `window_time` now accept nanosecond-precision timestamp columns, and the resulting window bounds preserve nanosecond precision.
### How was this patch tested?
New `DataFrameTimeWindowingSuite` and `DataFrameSessionWindowingSuite` tests for NTZ and LTZ nanosecond inputs, including value-level `checkAnswer` assertions for both (window bounds, session merging, and `window_time` at nanosecond granularity). All existing window/session tests pass.
### Was this patch authored or co-authored using generative AI tooling?
Authored with assistance by Claude Opus 4.8.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on PR #56951:
URL: https://github.com/apache/spark/pull/56951#issuecomment-6216630019
Hey @yadavay-amzn what is the status of this change? Did you get a chance to take a look at @HeartSaVioR comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57829][SQL] Support window, session_window and window_time over nanosecond-precision timestamps [spark]
Posted by "yadavay-amzn (via GitHub)" <gi...@apache.org>.
yadavay-amzn commented on PR #56951:
URL: https://github.com/apache/spark/pull/56951#issuecomment-4888018849
Following the review discussion on #56944 and #56984 (thanks @uros-b), this PR now carries the consolidated window-struct-over-nanos watermark handling: the CheckAnalysis window-struct guard accepts a nanosecond end field, and the statefulOperators eviction struct-branch extracts endFieldType via watermarkLiteral. That handling depends on the watermark infrastructure from #56944 (SPARK-57830), so this PR is stacked on it - the diff shows #56944's commits until it merges, after which I will rebase onto master. Added e2e tests: window() over a nanosecond event-time column + withWatermark, asserting nanosecond window bounds and state eviction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org