You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2026/06/26 19:35:28 UTC
(spark) branch branch-4.x updated: [SPARK-57303][SQL] Store-assignment and up-cast rules for nanosecond-precision timestamp types
This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 95bbc4007101 [SPARK-57303][SQL] Store-assignment and up-cast rules for nanosecond-precision timestamp types
95bbc4007101 is described below
commit 95bbc4007101801079a2c268bf50d0cd6ce059fb
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Fri Jun 26 21:35:02 2026 +0200
[SPARK-57303][SQL] Store-assignment and up-cast rules for nanosecond-precision timestamp types
### What changes were proposed in this pull request?
This PR defines a precision-safe store-assignment / up-cast contract for the whole LTZ/NTZ timestamp family - the microsecond types (`TIMESTAMP` / `TIMESTAMP_NTZ`) and their nanosecond-precision counterparts (`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in `[7, 9]`) - using a single notion of effective fractional-second precision (micros = 6, nanos = `p`).
For any ordered pair of timestamp-family types (including across the LTZ/NTZ boundary, which Spark already treats as a mutual up-cast for the micro types):
- target precision `>=` source precision: lossless widening -> up-cast (STRICT) and ANSI-store-assignable;
- target precision `<` source precision: lossy narrowing -> not an up-cast, blocked under ANSI so it can never silently truncate.
`DATE <-> nanos` is aligned to the micro `DATE <-> TIMESTAMP` behavior: `DATE -> nanos` is a lossless widening (up-cast + ANSI-store-assignable), while `nanos -> DATE` drops the time-of-day (not an up-cast, but still ANSI-store-assignable). LEGACY policy and explicit `CAST` are unchanged (they still truncate on narrowing). `TIME <-> timestamp` is unchanged and stays consistent with `TIME <-> micros` (never an up-cast, ANSI-store-assignable both ways).
Concretely:
- New shared `private[sql] object TimestampFamily` (`sql/api`) with `fractionalPrecision(dt): Option[Int]` plus `isLtz` / `isNtz`, reused across the rule sites (no type-hierarchy change, so no MiMa impact).
- `UpCastRule.canUpCast`: a single lossless-widening arm for the family (subsuming the existing `TimestampType <-> TimestampNTZType` cases), plus a generalized `DATE -> family` widening arm.
- `Cast.canANSIStoreAssign`: replaced the piecemeal per-subtask arms with one family narrowing block built on the shared helper, before the generic `DatetimeType` arm.
- `TypeCoercionHelper.findWiderDateTimeType`: refactored onto the shared helper (behavior-preserving) and updated the now-stale comment, since common-type resolution and the cast rules now agree on admissibility.
### Why are the changes needed?
Before this change, the nanosecond timestamp types fell through the generic `(_: DatetimeType, _: DatetimeType)` arm in `Cast.canANSIStoreAssign` (risking silent sub-microsecond truncation handled only narrowly), and they were absent from `UpCastRule.canUpCast`, so STRICT store assignment and up-cast resolution rejected even lossless widening. This PR gives the family a complete, precision-safe contract consistent with the microsecond precedent.
### Does this PR introduce _any_ user-facing change?
No. The nanosecond-precision timestamp types are unreleased (`Unstable`), so this only affects behavior within the unreleased branch.
### How was this patch tested?
- Updated the `SPARK-57293` / `SPARK-57490` / cross-family / micro-boundary contract tests in `CastSuiteBase` to the precision-safe widening model.
- Added a full-matrix predicate test over all 8 timestamp-family types asserting `canUpCast` and `canANSIStoreAssign` are true iff target precision `>=` source precision, plus `DATE` and `TIME` consistency anchors.
- Ran `CastSuite`, `CastWithAnsiOn/Off`, `TypeCoercionSuite`, `AnsiTypeCoercionSuite`, `DataTypeWriteCompatibilitySuite`, `V2WriteAnalysisSuite`, and `SQLQueryTestSuite` (cast / try_cast / nanos / typeCoercion) - all pass with no golden-file changes.
- Added coverage for two downstream consumers of `canUpCast` that the predicate-level tests do not reach: a `CastSuiteBase` test that `Cast.nullable`'s try-cast branch follows up-cast admissibility for the timestamp family (non-null preserved on widening, conservatively nullable on narrowing), and a new `GeneratedColumnExpressionSuite` asserting `GeneratedColumnExpression.validate` accepts a lossless widening generation expression and rejects a lossy narrowing.
### Note on scope
`canUpCast` / `canANSIStoreAssign` feed several consumers beyond up-cast resolution and store assignment (generated-column validation, subquery decorrelation, V2 expression pushdown, `Cast` try-cast nullability, and the Spark Connect `ArrowVectorReader` guard). The widening relaxation here is lossless and applies uniformly to all of them. One follow-up item: nanosecond timestamp types are not yet supported over Spark Connect (no `ConnectTypeOps` / vector reader), so `ArrowVectorReader [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56810 from MaxGekk/nanos-store-assignment.
Authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Max Gekk <ma...@gmail.com>
(cherry picked from commit 3791cb9068795d42d3bb5b4c4ea9b99563181085)
Signed-off-by: Max Gekk <ma...@gmail.com>
---
.../apache/spark/sql/types/TimestampFamily.scala | 49 +++++++
.../org/apache/spark/sql/types/UpCastRule.scala | 18 ++-
.../sql/catalyst/analysis/TypeCoercionHelper.scala | 33 ++---
.../spark/sql/catalyst/expressions/Cast.scala | 31 +----
.../sql/catalyst/expressions/CastSuiteBase.scala | 141 ++++++++++++++++-----
.../logical/GeneratedColumnExpressionSuite.scala | 50 ++++++++
6 files changed, 240 insertions(+), 82 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampFamily.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampFamily.scala
new file mode 100644
index 000000000000..c10670b14c3b
--- /dev/null
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampFamily.scala
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+/**
+ * Shared classification helpers for the LTZ/NTZ timestamp family: the microsecond types
+ * ([[TimestampType]] / [[TimestampNTZType]]) and their nanosecond-precision counterparts
+ * ([[TimestampLTZNanosType]] / [[TimestampNTZNanosType]]). Centralizes the notion of effective
+ * fractional-second precision and time-zone family so that up-cast resolution ([[UpCastRule]]),
+ * ANSI store assignment, and common-type resolution all agree.
+ */
+private[sql] object TimestampFamily {
+
+ /**
+ * The effective fractional-second precision of a timestamp-family type, or [[None]] for types
+ * that are not on the timestamp fractional-precision axis (DATE, TIME, and everything else).
+ * The microsecond types [[TimestampType]] / [[TimestampNTZType]] have precision 6; the
+ * nanosecond types carry their own precision `p` in [7, 9].
+ */
+ def fractionalPrecision(dt: DataType): Option[Int] = dt match {
+ case TimestampType | TimestampNTZType => Some(6)
+ case t: TimestampLTZNanosType => Some(t.precision)
+ case t: TimestampNTZNanosType => Some(t.precision)
+ case _ => None
+ }
+
+ /** Whether `dt` is a local-time-zone (instant) timestamp: micro [[TimestampType]] or nanos. */
+ def isLtz(dt: DataType): Boolean =
+ dt.isInstanceOf[TimestampType] || dt.isInstanceOf[TimestampLTZNanosType]
+
+ /** Whether `dt` is a no-time-zone (local) timestamp: micro [[TimestampNTZType]] or nanos. */
+ def isNtz(dt: DataType): Boolean =
+ dt.isInstanceOf[TimestampNTZType] || dt.isInstanceOf[TimestampNTZNanosType]
+}
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/UpCastRule.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/UpCastRule.scala
index 6272cb03bd79..54de45f6eb8c 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/UpCastRule.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/UpCastRule.scala
@@ -36,10 +36,20 @@ private[sql] object UpCastRule {
case (from: NumericType, to: DecimalType) if to.isWiderThan(from) => true
case (from: DecimalType, to: NumericType) if from.isTighterThan(to) => true
case (f, t) if legalNumericPrecedence(f, t) => true
- case (DateType, TimestampType) => true
- case (DateType, TimestampNTZType) => true
- case (TimestampNTZType, TimestampType) => true
- case (TimestampType, TimestampNTZType) => true
+ // Widening DATE -> timestamp family (micro or nanos, LTZ or NTZ) is lossless; the reverse
+ // (timestamp -> DATE) drops the time-of-day and is not matched here, so it stays a non-up-cast.
+ case (DateType, t) if TimestampFamily.fractionalPrecision(t).isDefined => true
+ // Lossless widening within the timestamp family: target fractional-second precision >= source.
+ // Covers micros <-> nanos and the cross-family LTZ <-> NTZ pairs (mirroring how the micro
+ // TimestampType <-> TimestampNTZType pair is a mutual up-cast). Same-type equal precision is
+ // short-circuited by `from == to` above; cross-family equal precision (e.g. LTZ(7) <-> NTZ(7))
+ // is admitted here by the `<=`. The guard keeps non-timestamp pairs falling through to the
+ // cases below; lossy narrowing falls through to `case _ => false`.
+ case (f, t)
+ if TimestampFamily
+ .fractionalPrecision(f)
+ .exists(fp => TimestampFamily.fractionalPrecision(t).exists(fp <= _)) =>
+ true
case (s1: StringType, s2: StringType) => StringHelper.isMoreConstrained(s1, s2)
// TODO: allow upcast from int/double/decimal to char/varchar of sufficient length
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
index e6329e465e00..942a6be948d8 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercionHelper.scala
@@ -83,6 +83,7 @@ import org.apache.spark.sql.types.{
StringType,
StringTypeExpression,
StructType,
+ TimestampFamily,
TimestampLTZNanosType,
TimestampNTZNanosType,
TimestampNTZType,
@@ -264,27 +265,21 @@ abstract class TypeCoercionHelper {
// The (family, precision) pair then maps back to a concrete type: precision 6 yields the
// micro type, precision in [7, 9] yields the nanos type.
//
- // Note: this common-type resolution is intentionally more permissive than the nanosecond
- // conversion rules in Cast.canUpCast / Cast.canANSIStoreAssign, which keep cross-family and
- // DATE <-> nanos casts explicit-CAST-only while the nanos types are unreleased (SPARK-57323
- // etc.). Coercion here mirrors the microsecond precedent so that UNION / CASE / coalesce /
- // IN / comparison resolve a common type the same way they do for the micro families; the
- // stricter explicit-only stance is deliberately scoped to up-cast and store assignment, not
- // to common-type resolution.
+ // Note: common-type resolution here is symmetric and widens to the maximum precision, while
+ // Cast.canUpCast / Cast.canANSIStoreAssign are directional (they block lossy narrowing). Both
+ // now agree on admissibility across the timestamp family -- including the cross-family
+ // LTZ <-> NTZ pairs and DATE <-> nanos (SPARK-57303) -- mirroring the microsecond precedent
+ // so that UNION / CASE / coalesce / IN / comparison resolve a common type the same way they
+ // do for the micro families.
case _ =>
- // Fractional-seconds precision of the microsecond timestamp types; the nanos types carry
- // 7-9. DATE has no time component and is treated as the micro precision so that
- // DATE <-> micro widens to the micro type and DATE <-> nanos to the nanos type.
+ // Fractional-seconds precision of the timestamp family (micros: 6, nanos: 7-9). DATE has no
+ // time component and is treated as the micro precision (getOrElse) so that DATE <-> micro
+ // widens to the micro type and DATE <-> nanos to the nanos type.
val MicrosPrecision = 6
- def isLtz(d: DatetimeType): Boolean =
- d.isInstanceOf[TimestampType] || d.isInstanceOf[TimestampLTZNanosType]
- def isNtz(d: DatetimeType): Boolean =
- d.isInstanceOf[TimestampNTZType] || d.isInstanceOf[TimestampNTZNanosType]
- def precisionOf(d: DatetimeType): Int = d match {
- case t: TimestampLTZNanosType => t.precision
- case t: TimestampNTZNanosType => t.precision
- case _ => MicrosPrecision // DateType / TimestampType / TimestampNTZType
- }
+ def isLtz(d: DatetimeType): Boolean = TimestampFamily.isLtz(d)
+ def isNtz(d: DatetimeType): Boolean = TimestampFamily.isNtz(d)
+ def precisionOf(d: DatetimeType): Int =
+ TimestampFamily.fractionalPrecision(d).getOrElse(MicrosPrecision)
// Beyond TimeType (handled above), the only datetime types are DATE and the micro/nanos
// timestamp families. Guard so that a future DatetimeType subtype fails fast here instead
// of being silently mis-widened (treated as a family-neutral precision-6 type and folded
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
index 346047a8ba82..96ebe62b76ad 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
@@ -500,30 +500,13 @@ object Cast extends QueryErrorsBase {
case (_: NumericType, _: NumericType) => true
case (_: AtomicType, _: StringType) => true
case (_: CalendarIntervalType, _: StringType) => true
- // SPARK-57490: same-family cross-precision nanosecond casts: widening (e.g. TIMESTAMP_NTZ(7) ->
- // TIMESTAMP_NTZ(9)) is lossless and allowed as a silent store assignment, while narrowing
- // (e.g. (9) -> (7)) drops sub-microsecond digits and stays explicit-only. Equal precision is
- // handled by the `from == to` short-circuit above; micros -> nanos widening (e.g. TIMESTAMP_NTZ
- // -> TIMESTAMP_NTZ(9)) is lossless and falls to the catch-all below.
- case (f: TimestampNTZNanosType, t: TimestampNTZNanosType) => f.precision <= t.precision
- case (f: TimestampLTZNanosType, t: TimestampLTZNanosType) => f.precision <= t.precision
- // SPARK-57323: DATE <-> nanosecond-precision timestamp requires an explicit CAST in both
- // directions (nanos -> DATE drops fields; DATE -> nanos is lossless but kept explicit-only
- // while the nanos types are unreleased). Stricter than micro DATE <-> TIMESTAMP[_NTZ], which
- // the catch-all below allows.
- case (DateType, _: AnyTimestampNanoType) => false
- case (_: AnyTimestampNanoType, DateType) => false
- // SPARK-57293/57511: narrowing any nanosecond timestamp to a microsecond timestamp drops the
- // sub-microsecond digits, and cross-family casts additionally reinterpret the value against the
- // session time zone; both stay explicit-only rather than silent store assignments while the
- // nanos types are unreleased. This covers same-family narrowing (nanos -> micro), cross-family
- // nanos <-> nanos, and the mixed micro/nanos pairs at the precision-6 boundary; everything
- // matched here is explicit-only. The all-micro TIMESTAMP <-> TIMESTAMP_NTZ pair and micros ->
- // nanos same-family widening stay store-assignable via the catch-all below.
- case (_: AnyTimestampNanoType, t) if AnyTimestampType.acceptsType(t) => false
- case (TimestampType, _: TimestampNTZNanosType) => false
- case (TimestampNTZType, _: TimestampLTZNanosType) => false
- case (_: AnyTimestampNanoType, _: AnyTimestampNanoType) => false
+ // SPARK-57303: block lossy narrowing across the whole timestamp family (LTZ/NTZ, micros and
+ // nanos, including the cross-family LTZ <-> NTZ pairs) so store assignment never silently drops
+ // sub-microsecond digits. Lossless widening, equal precision, and DATE <-> timestamp (DATE has
+ // no fractional precision, so it never matches here) all fall through to the DatetimeType arm
+ // below, mirroring the micro TIMESTAMP <-> TIMESTAMP_NTZ behavior.
+ case (f, t) if TimestampFamily.fractionalPrecision(f)
+ .exists(fp => TimestampFamily.fractionalPrecision(t).exists(fp > _)) => false
// SPARK-57585: widening a TIME(p) to a larger precision is lossless and allowed as a silent
// store assignment, while narrowing (e.g. TIME(6) -> TIME(3)) drops fractional-seconds digits
// and stays explicit-CAST-only. Equal precision is handled by the `from == to` short-circuit.
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 23d5783155fd..8f37f17c1069 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -754,12 +754,14 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
- test("SPARK-57293: nanos<->micros store-assignment and up-cast contract") {
+ test("SPARK-57303: nanos<->micros store-assignment and up-cast contract") {
foreachNanosPrecision { p =>
- // Explicit-only: neither direction is an up-cast, so STRICT store assignment rejects both.
- assert(!Cast.canUpCast(TimestampNTZType, TimestampNTZNanosType(p)))
+ // Lossless widening micros -> nanos(p) is an up-cast, mirroring the micro precedent where a
+ // lower-precision timestamp widens to a higher-precision one.
+ assert(Cast.canUpCast(TimestampNTZType, TimestampNTZNanosType(p)))
+ assert(Cast.canUpCast(TimestampType, TimestampLTZNanosType(p)))
+ // Lossy narrowing nanos(p) -> micros drops sub-microsecond digits, so it is not an up-cast.
assert(!Cast.canUpCast(TimestampNTZNanosType(p), TimestampNTZType))
- assert(!Cast.canUpCast(TimestampType, TimestampLTZNanosType(p)))
assert(!Cast.canUpCast(TimestampLTZNanosType(p), TimestampType))
// ANSI store assignment allows the lossless widening micros -> nanos(p) ...
@@ -769,18 +771,17 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
assert(!Cast.canANSIStoreAssign(TimestampNTZNanosType(p), TimestampNTZType))
assert(!Cast.canANSIStoreAssign(TimestampLTZNanosType(p), TimestampType))
- // SPARK-57323: DATE <-> nanos requires an explicit CAST in both directions, so STRICT
- // store assignment and ANSI store assignment both reject it. STRICT goes through
- // Cast.canUpCast, so the assertions below also guard against a future blanket datetime arm
- // in UpCastRule silently turning this into a safe store assignment.
- assert(!Cast.canUpCast(DateType, TimestampNTZNanosType(p)))
+ // SPARK-57303: DATE <-> nanos mirrors micro DATE <-> TIMESTAMP[_NTZ]. The lossless widening
+ // DATE -> nanos is an up-cast and ANSI-store-assignable; the lossy nanos -> DATE drops the
+ // time-of-day, so it is not an up-cast but is still ANSI-store-assignable.
+ assert(Cast.canUpCast(DateType, TimestampNTZNanosType(p)))
assert(!Cast.canUpCast(TimestampNTZNanosType(p), DateType))
- assert(!Cast.canUpCast(DateType, TimestampLTZNanosType(p)))
+ assert(Cast.canUpCast(DateType, TimestampLTZNanosType(p)))
assert(!Cast.canUpCast(TimestampLTZNanosType(p), DateType))
- assert(!Cast.canANSIStoreAssign(DateType, TimestampNTZNanosType(p)))
- assert(!Cast.canANSIStoreAssign(TimestampNTZNanosType(p), DateType))
- assert(!Cast.canANSIStoreAssign(DateType, TimestampLTZNanosType(p)))
- assert(!Cast.canANSIStoreAssign(TimestampLTZNanosType(p), DateType))
+ assert(Cast.canANSIStoreAssign(DateType, TimestampNTZNanosType(p)))
+ assert(Cast.canANSIStoreAssign(TimestampNTZNanosType(p), DateType))
+ assert(Cast.canANSIStoreAssign(DateType, TimestampLTZNanosType(p)))
+ assert(Cast.canANSIStoreAssign(TimestampLTZNanosType(p), DateType))
}
}
@@ -789,10 +790,11 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
p1 <- TimestampNTZNanosType.MIN_PRECISION to TimestampNTZNanosType.MAX_PRECISION
p2 <- TimestampNTZNanosType.MIN_PRECISION to TimestampNTZNanosType.MAX_PRECISION
} {
- // Cross-precision nanos casts are never up-casts (only equal precision is, via from == to),
- // matching the micros <-> nanos precedent above; STRICT store assignment rejects them.
- assert(Cast.canUpCast(TimestampNTZNanosType(p1), TimestampNTZNanosType(p2)) == (p1 == p2))
- assert(Cast.canUpCast(TimestampLTZNanosType(p1), TimestampLTZNanosType(p2)) == (p1 == p2))
+ // Lossless widening (p1 <= p2) is an up-cast; lossy narrowing (p1 > p2) is not, matching the
+ // micros <-> nanos precedent above. STRICT store assignment accepts widening, rejects
+ // narrowing.
+ assert(Cast.canUpCast(TimestampNTZNanosType(p1), TimestampNTZNanosType(p2)) == (p1 <= p2))
+ assert(Cast.canUpCast(TimestampLTZNanosType(p1), TimestampLTZNanosType(p2)) == (p1 <= p2))
// ANSI store assignment allows lossless widening (p1 <= p2) and equal precision, but blocks
// lossy narrowing (p1 > p2) to avoid silently dropping sub-microsecond digits.
assert(Cast.canANSIStoreAssign(TimestampNTZNanosType(p1), TimestampNTZNanosType(p2)) ==
@@ -829,12 +831,13 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
assert(Cast.canCast(ntz, ltz))
assert(Cast.canAnsiCast(ltz, ntz))
assert(Cast.canAnsiCast(ntz, ltz))
- // The cross-family reinterpretation against the session zone is never a safe up-cast.
- assert(!Cast.canUpCast(ltz, ntz))
- assert(!Cast.canUpCast(ntz, ltz))
- // They stay explicit-only: never silent store assignments (mirroring the other nanos casts).
- assert(!Cast.canANSIStoreAssign(ltz, ntz))
- assert(!Cast.canANSIStoreAssign(ntz, ltz))
+ // SPARK-57303: the cross-family LTZ <-> NTZ pair is treated on the precision axis like the
+ // micro TIMESTAMP <-> TIMESTAMP_NTZ pair: widening (target precision >= source) is an up-cast
+ // and ANSI-store-assignable, while lossy narrowing is neither.
+ assert(Cast.canUpCast(ltz, ntz) == (p <= q))
+ assert(Cast.canUpCast(ntz, ltz) == (q <= p))
+ assert(Cast.canANSIStoreAssign(ltz, ntz) == (p <= q))
+ assert(Cast.canANSIStoreAssign(ntz, ltz) == (q <= p))
// The conversion depends on the session time zone in both directions.
assert(Cast.needsTimeZone(ltz, ntz))
assert(Cast.needsTimeZone(ntz, ltz))
@@ -847,31 +850,99 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
test("cross-family nanos cast: micro boundary (precision 6) admissibility and store contract") {
// TIMESTAMP_LTZ(6) = TIMESTAMP and TIMESTAMP_NTZ(6) = TIMESTAMP_NTZ, so the precision-6
- // cross-family casts are the mixed micro/nanos pairs covered here.
+ // cross-family casts are the mixed micro/nanos pairs covered here. Each entry pairs a cast with
+ // whether it is a lossless widening (target precision >= source); p in [7, 9] always widens
+ // from the micro side (6) and always narrows to it.
foreachNanosPrecision { p =>
val pairs = Seq(
- (TimestampType: DataType, TimestampNTZNanosType(p): DataType), // LTZ(6) -> NTZ(p)
- (TimestampNTZNanosType(p): DataType, TimestampType: DataType), // NTZ(p) -> LTZ(6)
- (TimestampNTZType: DataType, TimestampLTZNanosType(p): DataType),// NTZ(6) -> LTZ(p)
- (TimestampLTZNanosType(p): DataType, TimestampNTZType: DataType))// LTZ(p) -> NTZ(6)
- pairs.foreach { case (from, to) =>
- // Explicit casts are allowed (ANSI and non-ANSI), but are never safe up-casts and never
- // silent store assignments, and they depend on the session time zone.
+ (TimestampType: DataType, TimestampNTZNanosType(p): DataType, true), // LTZ(6) -> NTZ(p)
+ (TimestampNTZNanosType(p): DataType, TimestampType: DataType, false), // NTZ(p) -> LTZ(6)
+ (TimestampNTZType: DataType, TimestampLTZNanosType(p): DataType, true),// NTZ(6) -> LTZ(p)
+ (TimestampLTZNanosType(p): DataType, TimestampNTZType: DataType, false))// LTZ(p) -> NTZ(6)
+ pairs.foreach { case (from, to, widening) =>
+ // Explicit casts are allowed (ANSI and non-ANSI) and depend on the session time zone.
assert(Cast.canCast(from, to))
assert(Cast.canAnsiCast(from, to))
- assert(!Cast.canUpCast(from, to))
- assert(!Cast.canANSIStoreAssign(from, to))
+ // SPARK-57303: widening is an up-cast and store-assignable; narrowing is neither.
+ assert(Cast.canUpCast(from, to) == widening)
+ assert(Cast.canANSIStoreAssign(from, to) == widening)
assert(Cast.needsTimeZone(from, to))
// Null-safe like the micro TIMESTAMP <-> TIMESTAMP_NTZ pair.
assert(!Cast.forceNullable(from, to))
}
}
// Sanity: the all-micro TIMESTAMP <-> TIMESTAMP_NTZ pair (precision 6 <-> 6) stays a silent
- // store assignment, unlike the mixed micro/nanos pairs above.
+ // store assignment (equal precision).
assert(Cast.canANSIStoreAssign(TimestampType, TimestampNTZType))
assert(Cast.canANSIStoreAssign(TimestampNTZType, TimestampType))
}
+ test("SPARK-57303: full timestamp-family up-cast and store-assignment precision matrix") {
+ // The micro/nanos LTZ/NTZ timestamp types with their effective fractional-second precision
+ // (micros: 6, nanos: 7-9), across both time-zone families.
+ val tsTypes: Seq[DataType] =
+ Seq(TimestampType, TimestampNTZType) ++
+ (TimestampLTZNanosType.MIN_PRECISION to TimestampLTZNanosType.MAX_PRECISION).flatMap { p =>
+ Seq(TimestampLTZNanosType(p), TimestampNTZNanosType(p))
+ }
+ def precisionOf(dt: DataType): Int = dt match {
+ case t: TimestampLTZNanosType => t.precision
+ case t: TimestampNTZNanosType => t.precision
+ case _ => 6
+ }
+ // For every ordered pair, canUpCast and canANSIStoreAssign are true iff the target precision is
+ // >= the source precision (lossless widening or equal precision), false for lossy narrowing.
+ for {
+ from <- tsTypes
+ to <- tsTypes
+ } {
+ val widening = precisionOf(from) <= precisionOf(to)
+ withClue(s"$from -> $to: ") {
+ assert(Cast.canUpCast(from, to) == widening)
+ assert(Cast.canANSIStoreAssign(from, to) == widening)
+ }
+ }
+
+ // DATE anchors (micros and nanos): DATE -> ts is a lossless widening (up-cast + store-assign);
+ // ts -> DATE drops the time-of-day (not an up-cast) but stays ANSI-store-assignable.
+ tsTypes.foreach { ts =>
+ assert(Cast.canUpCast(DateType, ts), s"DATE -> $ts should be an up-cast")
+ assert(Cast.canANSIStoreAssign(DateType, ts), s"DATE -> $ts should be store-assignable")
+ assert(!Cast.canUpCast(ts, DateType), s"$ts -> DATE should not be an up-cast")
+ assert(Cast.canANSIStoreAssign(ts, DateType), s"$ts -> DATE should be store-assignable")
+ }
+
+ // TIME anchors: TIME is intentionally outside the timestamp family, so TIME <-> ts matches the
+ // micro TIME <-> TIMESTAMP behavior - never an up-cast, but ANSI-store-assignable both ways.
+ for {
+ tq <- TimeType.MIN_PRECISION to TimeType.MAX_PRECISION
+ ts <- tsTypes
+ } {
+ val time = TimeType(tq)
+ assert(!Cast.canUpCast(time, ts), s"$time -> $ts should not be an up-cast")
+ assert(!Cast.canUpCast(ts, time), s"$ts -> $time should not be an up-cast")
+ assert(Cast.canANSIStoreAssign(time, ts), s"$time -> $ts should be store-assignable")
+ assert(Cast.canANSIStoreAssign(ts, time), s"$ts -> $time should be store-assignable")
+ }
+ }
+
+ test("SPARK-57303: try-cast nullability follows up-cast admissibility for the timestamp family") {
+ // `Cast.nullable`'s try-cast branch keys on `Cast.canUpCast`: an up-cast (lossless widening
+ // within the timestamp family, or DATE -> ts) never fails, so a non-null child stays non-null;
+ // a lossy narrowing is not an up-cast, so the try-cast is conservatively nullable.
+ def tryCast(from: DataType, to: DataType): Cast =
+ Cast(AttributeReference("c", from, nullable = false)(), to, evalMode = EvalMode.TRY)
+ foreachNanosPrecision { p =>
+ // Lossless widening micros -> nanos(p) and DATE -> nanos(p): non-null child stays non-null.
+ assert(!tryCast(TimestampNTZType, TimestampNTZNanosType(p)).nullable)
+ assert(!tryCast(TimestampType, TimestampLTZNanosType(p)).nullable)
+ assert(!tryCast(DateType, TimestampNTZNanosType(p)).nullable)
+ // Lossy narrowing nanos(p) -> micros is not an up-cast, so the try-cast is nullable.
+ assert(tryCast(TimestampNTZNanosType(p), TimestampNTZType).nullable)
+ assert(tryCast(TimestampLTZNanosType(p), TimestampType).nullable)
+ }
+ }
+
test("SPARK-40389: canUpCast: return false if casting decimal to integral types can cause" +
" overflow") {
Seq(ByteType, ShortType, IntegerType, LongType).foreach { integralType =>
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
new file mode 100644
index 000000000000..de11b78452cc
--- /dev/null
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/plans/logical/GeneratedColumnExpressionSuite.scala
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType, TimestampNTZNanosType}
+import org.apache.spark.sql.types.{TimestampNTZType, TimestampType}
+
+class GeneratedColumnExpressionSuite extends SparkFunSuite {
+
+ private def genExpr(childType: DataType): GeneratedColumnExpression =
+ GeneratedColumnExpression(Literal.create(null, childType), "<gen expr>")
+
+ test("SPARK-57303: validate accepts a lossless widening to a nanosecond timestamp column") {
+ // The generation expression's type is up-castable to the column type, so validate() succeeds:
+ // micros -> nanos is a lossless widening up-cast (Cast.canUpCast).
+ (TimestampNTZNanosType.MIN_PRECISION to TimestampNTZNanosType.MAX_PRECISION).foreach { p =>
+ genExpr(TimestampNTZType).validate("c", TimestampNTZNanosType(p), allColumns = Seq.empty)
+ genExpr(TimestampType).validate("c", TimestampLTZNanosType(p), allColumns = Seq.empty)
+ }
+ }
+
+ test("SPARK-57303: validate rejects a lossy narrowing from a nanosecond timestamp column") {
+ // nanos -> micros drops sub-microsecond digits and is not an up-cast, so validate() rejects it.
+ (TimestampNTZNanosType.MIN_PRECISION to TimestampNTZNanosType.MAX_PRECISION).foreach { p =>
+ val ex = intercept[AnalysisException] {
+ genExpr(TimestampNTZNanosType(p)).validate("c", TimestampNTZType, allColumns = Seq.empty)
+ }
+ assert(ex.getCondition == "UNSUPPORTED_EXPRESSION_GENERATED_COLUMN")
+ assert(ex.getMessageParameters.get("reason").contains("incompatible with column data type"))
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org