You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2026/06/15 14:04:36 UTC
[PR] [SPARK-57469][SQL] Support date field functions on nanosecond-precision timestamps in ANSI mode [spark]
MaxGekk opened a new pull request, #56518:
URL: https://github.com/apache/spark/pull/56518
### What changes were proposed in this pull request?
This PR makes the date field extraction functions (`year`, `quarter`, `month`, `day`/`dayofmonth`, `dayofyear`, `dayofweek`, `weekday`, `weekofyear`, `yearofweek`) and the transitive `EXTRACT` / `date_part` date components work on the nanosecond-precision timestamp types `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (`p` in `[7, 9]`) in ANSI mode.
- Add `AnyTimestampNanoType` (`AbstractDataType`) and `AnyTimestampNanoTypeExpression` (expression extractor) matching `TimestampLTZNanosType` / `TimestampNTZNanosType`, mirroring the existing micro abstractions `AnyTimestampType` / `AnyTimestampTypeExpression`.
- Extend `AnsiGetDateFieldOperationsTypeCoercion` to also match nanos-timestamp children of `GetDateField` and cast them to `DATE`, exactly as it already does for micro timestamps. This deliberately does not widen `AnyTimestampType` (also used as an "accept-as-is, no cast" `inputTypes` gate on micro-only expressions), and keeps `Cast.canANSIStoreAssign` / `Cast.canUpCast` strict for `DATE` <-> nanos (the cast is inserted explicitly by the field-extraction rule).
- Add nanosecond examples to the date field functions' `@ExpressionDescription`.
No `EXTRACT`-specific change is needed: `EXTRACT(field FROM source)` is a `RuntimeReplaceable` that rewrites via `DatePart.parseExtractField` to the same `GetDateField` expressions, so once the `GetDateField` coercion is fixed, `extract(year from nanos_ts)` and `date_part('year', nanos_ts)` work transitively in both ANSI and non-ANSI modes. Time-of-day fields (`HOUR`/`MINUTE`/`SECOND`) were already handled by SPARK-57340.
This is a sub-task of [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822) (SPIP: Timestamps with nanosecond precision) and builds on [SPARK-57323](https://issues.apache.org/jira/browse/SPARK-57323) (DATE <-> nanos casts).
### Why are the changes needed?
In ANSI mode (the default since Spark 4.0) the date field functions fail analysis with `DATATYPE_MISMATCH` on nanosecond timestamps. For example:
```sql
SET spark.sql.timestampNanosTypes.enabled=true;
SELECT year(TIMESTAMP_NTZ '2020-01-01 12:30:15.123456789'::timestamp_ntz(9));
```
fails, because the generic ANSI implicit-cast rule defers to `Cast.canANSIStoreAssign` (which returns false for nanos -> DATE by design) and the dedicated `AnsiGetDateFieldOperationsTypeCoercion` rule matched only the microsecond timestamp types. In non-ANSI mode the functions already work via the blanket `(_: DatetimeType, _: DatetimeType)` implicit-cast arm.
### Does this PR introduce _any_ user-facing change?
Yes. With `spark.sql.timestampNanosTypes.enabled=true`, date field functions and the `EXTRACT` / `date_part` date components now work on `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` in ANSI mode. Previously they failed analysis with `DATATYPE_MISMATCH`; they already worked in non-ANSI mode. The nanosecond timestamp types are a preview feature (disabled by default), so there is no change for released behavior.
### How was this patch tested?
- New unit tests in `TimestampNanosFunctionsSuiteBase` (run with ANSI mode on and off), covering the function form, the `EXTRACT` / `date_part` form, and the `functions.*` Column API, over a spread of values: leap day, ISO-week and quarter boundaries, pre-epoch dates, varied precisions (7/8/9) and fractions, LTZ time-zone date shifts, and NULLs.
- New golden-file queries appended to `timestamp-ltz-nanos.sql` / `timestamp-ntz-nanos.sql` with regenerated `.sql.out` files.
- `ExpressionInfoSuite` validates the new `@ExpressionDescription` examples.
- `./dev/scalastyle` and scalafmt pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57469][SQL] Support date field functions on nanosecond-precision timestamps in ANSI mode [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56518:
URL: https://github.com/apache/spark/pull/56518#issuecomment-4713260107
Merging to master/4.x. Thank you, @HyukjinKwon for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57469][SQL] Support date field functions on nanosecond-precision timestamps in ANSI mode [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56518: [SPARK-57469][SQL] Support date field functions on nanosecond-precision timestamps in ANSI mode
URL: https://github.com/apache/spark/pull/56518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org