You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2026/05/24 20:21:18 UTC
[PR] [WIP][SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
MaxGekk opened a new pull request, #56094:
URL: https://github.com/apache/spark/pull/56094
### What changes were proposed in this pull request?
This PR completes [SPARK-56969](https://issues.apache.org/jira/browse/SPARK-56969) on top of the parser gating added in [SPARK-56965](https://issues.apache.org/jira/browse/SPARK-56965) / [#56041](https://github.com/apache/spark/pull/56041).
- Extend `TypeUtils.failUnsupportedDataType` to recursively reject `TimestampNTZNanosType` and `TimestampLTZNanosType` when `spark.sql.timestampNanosTypes.enabled` is off.
- Add `UNSUPPORTED_TIMESTAMP_NANOS_TYPE` with a message naming the conf key.
- Expand the SQLConf doc for `spark.sql.timestampNanosTypes.enabled` and align its default with `Utils.isTesting` (mirroring `spark.sql.timeType.enabled`).
- Add a short enablement note for preview nanos timestamp types in `docs/sql-ref-datatypes.md`.
Part of SPIP [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822).
### Why are the changes needed?
Parser and JSON entry points already gate parameterized nanos timestamp types behind `spark.sql.timestampNanosTypes.enabled`, but analyzed schemas/plans could still surface these types through other paths (for example `CREATE TABLE`, connectors, or programmatic schemas) before downstream execution support is ready.
Analysis-time gating closes that gap and keeps behavior consistent with the existing preview flag.
### Does this PR introduce _any_ user-facing change?
Yes.
- When `spark.sql.timestampNanosTypes.enabled` is off, analyzed schemas/plans containing `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` with `p` in `[7, 9]` now fail with `UNSUPPORTED_TIMESTAMP_NANOS_TYPE`.
- The conf default is now `Utils.isTesting` instead of always `false`, so tests enable the preview by default while production remains off.
- `docs/sql-ref-datatypes.md` documents how to enable the preview feature.
Unparameterized `TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` behavior is unchanged.
Example:
```sql
SET spark.sql.timestampNanosTypes.enabled=false;
CREATE TABLE t (c TIMESTAMP_NTZ(9));
-- UNSUPPORTED_TIMESTAMP_NANOS_TYPE: ... Set "spark.sql.timestampNanosTypes.enabled" to "true" ...
```
### How was this patch tested?
Added/updated unit tests:
- TypeUtilsSuite: default conf behavior, analysis gating on/off, microsecond types unaffected
- DataTypeParserSuite: explicitly disable conf when testing parser rejection
- DataTypeSuite: explicitly disable conf when testing JSON rejection
Existing nanos parser/JSON tests continue to pass with the conf enabled.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Auto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56094:
URL: https://github.com/apache/spark/pull/56094#issuecomment-4544273861
Waiting for @stevomitric PR https://github.com/apache/spark/pull/56112 be merged as soon as CI passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #56094: [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled`
URL: https://github.com/apache/spark/pull/56094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56094:
URL: https://github.com/apache/spark/pull/56094#discussion_r3298544651
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -4832,6 +4832,12 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
messageParameters = Map.empty)
}
+ def unsupportedTimestampNanosTypeError(): Throwable = {
+ new AnalysisException(
Review Comment:
It makes sense. Reused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on code in PR #56094:
URL: https://github.com/apache/spark/pull/56094#discussion_r3297144737
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala:
##########
@@ -4832,6 +4832,12 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
messageParameters = Map.empty)
}
+ def unsupportedTimestampNanosTypeError(): Throwable = {
+ new AnalysisException(
Review Comment:
Why not reuse `FEATURE_NOT_ENABLED` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56094:
URL: https://github.com/apache/spark/pull/56094#issuecomment-4532586722
@stevomitric Please, have a look at the PR. It is related to your changes in https://github.com/apache/spark/pull/56041
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56094:
URL: https://github.com/apache/spark/pull/56094#issuecomment-4532013263
@dongjoon-hyun The PR is ready for review. I removed the `[WIP]` tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #56094:
URL: https://github.com/apache/spark/pull/56094#discussion_r3296569215
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -647,15 +647,19 @@ object SQLConf {
val TIMESTAMP_NANOS_TYPES_ENABLED =
buildConf("spark.sql.timestampNanosTypes.enabled")
.internal()
- .doc("When true, the parameterized nanosecond-precision timestamp types " +
- "TIMESTAMP_NTZ(p) / TIMESTAMP_LTZ(p) for p in [7, 9] are recognized as " +
- "Spark SQL data types at user-facing entry points. Default is false because " +
- "downstream execution paths (Cast, PhysicalDataType, AnyTimestampType, encoders, " +
- "Connect proto) are not yet wired for these types. See SPARK-56822.")
+ .doc("When true, allows nanosecond-capable timestamp types TIMESTAMP_NTZ(p) and " +
+ "TIMESTAMP_LTZ(p) with fractional seconds precision p in [7, 9] at user-facing " +
+ "entry points, including the SQL parser, schemas, and analyzed plans. This is a " +
+ "preview feature under SPARK-56822 and may change in future releases. The default is " +
+ "false in production; tests enable it by default via Utils.isTesting. " +
+ "Unparameterized TIMESTAMP, TIMESTAMP_NTZ, and TIMESTAMP_LTZ remain microsecond " +
+ "types. Enabling this flag does not guarantee full SQL support: casts, Parquet read, " +
+ "typed literals, and other operations may still fail until their respective features " +
+ "are implemented.")
.version("4.2.0")
Review Comment:
@HyukjinKwon @cloud-fan @dongjoon-hyun I merged this config to master, will it be released in 4.2.0? Should I merge new features to branch-4.x?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56969][SQL] Enhance the SQL config `spark.sql.timestampNanosTypes.enabled` [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #56094:
URL: https://github.com/apache/spark/pull/56094#issuecomment-4545804731
Merging to master/4.x. Thank you, @stevomitric @dongjoon-hyun for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org