[jira] [Resolved] (SPARK-56969) Add SQLConf preview flag for nanosecond-capable timestamp types

["Max Gekk (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-56969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Max Gekk resolved SPARK-56969.
------------------------------
    Fix Version/s: 4.3.0
       Resolution: Fixed

Issue resolved by pull request 56094
[https://github.com/apache/spark/pull/56094]

> Add SQLConf preview flag for nanosecond-capable timestamp types
> ---------------------------------------------------------------
>
>                 Key: SPARK-56969
>                 URL: https://issues.apache.org/jira/browse/SPARK-56969
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>    Affects Versions: 4.3.0
>            Reporter: Max Gekk
>            Assignee: Max Gekk
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.3.0
>
>
> h2. Summary
> Introduce a preview SQL configuration that gates use of {{TimestampNTZNanosType(p)}} and {{TimestampLTZNanosType(p)}} ({{p}} in [7, 9]) added in [SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876]. The flag is *off by default* in production (default {{true}} under {{Utils.isTesting}}, following {{spark.sql.timeType.enabled}} / {{TIME_TYPE_ENABLED}}).
> h2. Background
> Logical types and parser support exist ([SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876], [SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965]). Physical row storage lands in [SPARK-56981|https://issues.apache.org/jira/browse/SPARK-56981]. Without gating, partially implemented nanos types could surface in schemas, casts, or connectors before the SPIP preview milestone is ready.
> {{TimeType}} precedent: {{spark.sql.timeType.enabled}} with {{TypeUtils.failUnsupportedDataType}} and {{UNSUPPORTED_TIME_TYPE}}.
> h2. Scope
> h3. 1. SQLConf implementation
> * New entry (name TBD; suggest {{spark.sql.timestampNanos.preview.enabled}}):
> ** {{.internal()}} — preview / unstable
> ** {{.booleanConf}} with {{.createWithDefault(Utils.isTesting)}} (mirror {{TIME_TYPE_ENABLED}})
> ** {{.version("4.2.0")}}
> ** {{.doc(...)}} — see *Documentation* below
> * Accessor on {{SQLConf}} (e.g. {{isTimestampNanosPreviewEnabled}})
> h3. 2. Analysis gating
> * Extend {{TypeUtils.failUnsupportedDataType}} to reject schemas/plans that *recursively* contain {{TimestampNTZNanosType}} or {{TimestampLTZNanosType}} when the flag is disabled
> * Error class {{UNSUPPORTED_TIMESTAMP_NANOS_TYPE}} (or single message covering both NTZ/LTZ) in {{error-conditions.json}} and {{QueryCompilationErrors}}, analogous to {{UNSUPPORTED_TIME_TYPE}}
> * Error message must name the conf key and how to enable preview (supports 12b discoverability)
> h3. 3. Binding policy
> * Register the conf in {{configs-without-binding-policy-exceptions}} if required (see {{spark.sql.timeType.enabled}} precedent)
> h3. 4. Documentation (sub-task 12b — same PR)
> *Audience:* operators and advanced users.
> * *{{SQLConf}} {{.doc()}} text* must cover:
> ** What enabling allows (nanos-capable {{TIMESTAMP_NTZ(p)}} / {{TIMESTAMP_LTZ(p)}}, {{p}} in 7–9, in schemas and analyzed plans)
> ** Preview / unstable status under [SPARK-56822|https://issues.apache.org/jira/browse/SPARK-56822]
> ** Default value and testing default via {{Utils.isTesting}}
> ** That unparameterized {{TIMESTAMP}} / {{TIMESTAMP_NTZ}} / {{TIMESTAMP_LTZ}} remain microsecond types
> ** What may still fail when enabled (casts, Parquet read, literals, etc. until their JIRAs land)
> * *{{docs/configuration.md}}* — add a row/section for the new key (follow existing {{spark.sql.*}} entries):
> ** Key name (exact string as implemented)
> ** Default, version (4.2.0), scope, preview disclaimer
> ** Cross-link to SPIP JIRA or future sql-ref when available
> * *Short enablement note* (minimal user discovery before full sql-ref pass in 12c):
> ** One paragraph in {{docs/sql-ref-datatypes.md}} *or* a note in {{docs/configuration.md}} intro pointing to the conf
> ** Example: {{SET spark.sql.timestampNanos.preview.enabled=true;}}
> ** Do *not* document capabilities that are not yet merged; list only what works at ship time
> h2. Tests
> * Unit: {{SQLConf}} entry exists; default matches {{Utils.isTesting}} pattern; readable via {{SQLConf.get}}
> * Unit: {{TypeUtils.failUnsupportedDataType}} throws when flag off and schema contains {{TimestampNTZNanosType(9)}} / {{TimestampLTZNanosType(9)}}; passes when flag on
> * Unit: error message references the conf key (documentation/discoverability)
> * Regression: existing {{DataTypeSuite}} / types without nanos types unaffected
> * Optional: {{AnalysisSuite}} or {{DDLParsingSuite}} — {{CREATE TABLE t (c TIMESTAMP_NTZ(9))}} fails with new error when preview off ([SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965] parser merged)
> h2. Acceptance criteria
> * With preview *disabled* (default outside tests), any analyzed schema or plan that recursively contains {{TimestampNTZNanosType}} or {{TimestampLTZNanosType}} fails with a clear error naming {{spark.sql.timestampNanos.preview.enabled}} (exact key as implemented)
> * With preview *enabled* (or in tests), {{failUnsupportedDataType}} does not block those types (downstream JIRAs may still fail for other reasons)
> * No change to {{TimestampType}}, {{TimestampNTZType}}, or zero-arg {{TIMESTAMP}} / {{TIMESTAMP_NTZ}} DDL semantics
> * {{SQLConf}} {{.doc()}} text is sufficient for generated configuration reference
> * {{docs/configuration.md}} documents the key with correct default, version, and enablement instructions
> * At least one public doc location (configuration.md and/or sql-ref-datatypes) lets users find how to turn on preview without reading source
> * Conf discoverable via {{SET spark.sql.timestampNanos.preview.enabled}}
> h2. Dependencies
> * Logical types: [SPARK-56876|https://issues.apache.org/jira/browse/SPARK-56876]
> * Parser: [SPARK-56965|https://issues.apache.org/jira/browse/SPARK-56965] (merged)
> * Recommended before first user-visible Parquet read or cast PRs
> * Physical rows: [SPARK-56981|https://issues.apache.org/jira/browse/SPARK-56981] (in progress — gating can land independently)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org