You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/10/20 20:15:46 UTC
(spark) branch branch-4.x updated: [SPARK-57838][SQL] Harden overflow and range handling for nanosecond timestamps
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ed7d7a2b933f [SPARK-57838][SQL] Harden overflow and range handling for nanosecond timestamps
ed7d7a2b933f is described below
commit ed7d7a2b933f543876e7d9a925a3f3e07fc38f7d
Author: Stevo Mitric <st...@gmail.com>
AuthorDate: Tue Oct 20 20:15:24 2026 +0000
[SPARK-57838][SQL] Harden overflow and range handling for nanosecond timestamps
### What changes were proposed in this pull request?
This PR hardens range and overflow handling for `TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)` (p in [7, 9]). The representable range does not change: it stays the same as for the microsecond types.
- `DateTimeUtils.timestampNanosToEpochNanos`, the int64 epoch-nanosecond pack used by the Parquet INT64, Avro and Arrow writers and by Parquet filter pushdown, is now exact down to the int64 minimum. `epochMicros * 1000` overflowed for `epochMicros = -9223372036854776` even when the final value fits, so `1677-09-21 00:12:43.145224192` through `.145224999` UTC were rejected with `DATETIME_OVERFLOW`, although the error message names `1677-09-21T00:12:43.145224192Z` as the supported minimum.
- The Arrow nanos writers still reject the exact int64 minimum, because pandas/numpy reserve it for `NaT`: writing it would turn a non-null value into null in `toPandas()`, Spark Connect and pandas UDF inputs. Their `DATETIME_OVERFLOW` error now states the Arrow lower bound, `1677-09-21T00:12:43.145224193Z`. Parquet and Avro accept the full int64 range.
- `ts + INTERVAL ...` on nanos timestamps (`timestampNanosAddDayTime` / `timestampNanosAddMonths`) raises `DATETIME_OVERFLOW` when the result leaves the epoch-microsecond range, instead of a raw `java.lang.ArithmeticException: long overflow`. The message renders both operands as SQL literals of their own types, for example `add INTERVAL '1' DAY to TIMESTAMP_NTZ '+294247-01-10 04:00:54.775807999'`, like `timestampadd`, which already wraps its overflow.
- The `unix_nanos` comment no longer claims `DECIMAL(21, 0)` holds every value: it covers the 0001-9999 calendar range, and values more than ~31,700 years from the epoch raise `NUMERIC_VALUE_OUT_OF_RANGE`.
- `docs/sql-ref-datatypes.md` documents the range of the nanos types, the narrower int64 epoch-nanosecond range of Parquet, Avro and Arrow (including the Arrow/pandas lower bound), and the overflow behavior.
- Boundary tests at the calendar limits (`0001-01-01 00:00:00`, `9999-12-31 23:59:59.999999999`), the int64 epoch-nanosecond limits (`1677-09-21 00:12:43.145224192`, `2262-04-11 23:47:16.854775807`) and the epoch-microsecond limits, for parse, format, cast, interval and `timestampadd` arithmetic, `unix_nanos`, and the Parquet/Avro/Arrow writes.
### Why are the changes needed?
The SPIP (SPARK-56822, Q6) lists range and overflow as a risk, with these mitigations: document the representable range, enforce bounds where values are narrowed, and test edge instants. Before this PR the int64 sinks rejected valid values at the int64 minimum, nanos interval overflow escaped as a raw JVM exception, and the range wasn't documented.
Parsing and casting already behave like the microsecond types, because they share the parser: years after 9999 are accepted, and a value outside the epoch-microsecond range gives `NULL`, or `CAST_INVALID_INPUT` under ANSI mode. This PR keeps that parity and does not add a 0001-9999 check, which would make the nanos types stricter than `TIMESTAMP`.
### Does this PR introduce _any_ user-facing change?
Yes, for the preview nanosecond timestamp types only, which are not in a released version. The microsecond types are unchanged.
- Writing a nanos value from `1677-09-21 00:12:43.145224192` to `1677-09-21 00:12:43.145224999` UTC to Parquet or Avro now succeeds, and from `.145224193` to Arrow. Before, it failed with `DATETIME_OVERFLOW`.
- `ts + INTERVAL ...` past the epoch-microsecond range (around year 294247) raises `DATETIME_OVERFLOW` instead of `java.lang.ArithmeticException: long overflow`.
### How was this patch tested?
- New catalyst tests in `DateTimeUtilsSuite`, `TimestampNanosParseSuite`, `CastSuiteBase` / `CastWithAnsiOnSuite` / `CastWithAnsiOffSuite`, and `DateExpressionsSuite`: the boundaries above for parse, format, cast (both ANSI modes), interval add, `timestampadd` and `unix_nanos`, interpreted and codegen, including the overflow message.
- New sink tests at the exact int64 limits and one nanosecond past them: `ParquetTimestampNanosSuite` (including a check that the boundary filters are pushed down), `TimestampNanosParquetOpsSuite`, `AvroSuite`, and `ArrowWriterSuite` (the int64 minimum fails, minimum + 1 ns round-trips).
- The touched suites, the `TimestampNanos*` suites and the nanos golden files in `SQLQueryTestSuite` pass, and scalastyle is clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59579 from stevomitric/stevomitric/nanos-range-overflow.
Authored-by: Stevo Mitric <st...@gmail.com>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit 7150b5dca94fb790f57e67c068513d858196db78)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
.../org/apache/spark/sql/avro/AvroSuite.scala | 42 ++++++--
docs/sql-ref-datatypes.md | 1 +
.../catalyst/expressions/datetimeExpressions.scala | 18 +++-
.../spark/sql/catalyst/util/DateTimeUtils.scala | 45 ++++++--
.../spark/sql/errors/QueryExecutionErrors.scala | 21 +++-
.../spark/sql/execution/arrow/ArrowWriter.scala | 39 +++----
.../sql/catalyst/expressions/CastSuiteBase.scala | 26 +++++
.../expressions/CastWithAnsiOffSuite.scala | 9 ++
.../catalyst/expressions/CastWithAnsiOnSuite.scala | 13 +++
.../expressions/DateExpressionsSuite.scala | 93 +++++++++++++++++
.../sql/catalyst/util/DateTimeUtilsSuite.scala | 115 +++++++++++++++++----
.../catalyst/util/TimestampNanosParseSuite.scala | 50 +++++++++
.../sql/execution/arrow/ArrowWriterSuite.scala | 36 +++++--
.../parquet/ParquetTimestampNanosSuite.scala | 63 +++++++++++
.../types/ops/TimestampNanosParquetOpsSuite.scala | 13 +++
15 files changed, 512 insertions(+), 72 deletions(-)
diff --git a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
index 46a1ea727e5a..5805dfd2bf85 100644
--- a/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
+++ b/connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
@@ -3638,9 +3638,7 @@ abstract class AvroSuite
SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
// Long.MaxValue epoch-nanoseconds = 2262-04-11T23:47:16.854775807Z, the largest instant the
- // INT64 epoch-nanos encoding can represent. The lower bound Long.MinValue is intentionally
- // not exercised: re-encoding its floored epoch-microseconds overflows Long (see
- // DateTimeUtilsSuite), so it is not writable.
+ // INT64 epoch-nanos encoding can represent.
val maxInstant = java.time.Instant.ofEpochSecond(9223372036L, 854775807L)
val maxLocal = LocalDateTime.ofEpochSecond(9223372036L, 854775807, java.time.ZoneOffset.UTC)
Seq(
@@ -3663,13 +3661,11 @@ abstract class AvroSuite
withSQLConf(
SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
- // The smallest *writable* instant: Long.MinValue epoch-nanoseconds floor to an epochMicros
- // whose re-encode overflows Long, so the practical minimum is the next whole microsecond up
- // (~1677-09-21), the symmetric counterpart of the maximum-instant test above.
- val minEpochNanos = (Long.MinValue / 1000L) * 1000L
+ // The smallest instant: Long.MinValue epoch-nanoseconds = 1677-09-21T00:12:43.145224192Z,
+ // the symmetric counterpart of the maximum-instant test above.
// Pre-epoch full-precision value: 1969-12-31T23:59:59.999999999Z (= -1 epoch-nanosecond),
// exercising the floor semantics end to end (mirrors the Parquet/ORC suites).
- Seq(-1L, minEpochNanos).foreach { epochNanos =>
+ Seq(-1L, Long.MinValue).foreach { epochNanos =>
val seconds = Math.floorDiv(epochNanos, 1000000000L)
val nanoOfSecond = Math.floorMod(epochNanos, 1000000000L).toInt
val instant = java.time.Instant.ofEpochSecond(seconds, nanoOfSecond.toLong)
@@ -3692,6 +3688,36 @@ abstract class AvroSuite
}
}
+ test("SPARK-57838: writing one nanosecond past the int64 epoch-nanos range fails loudly") {
+ withSQLConf(
+ SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true",
+ SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ Seq(
+ "2262-04-11T23:47:16.854775808", "1677-09-21T00:12:43.145224191").foreach { ts =>
+ val local = LocalDateTime.parse(ts)
+ Seq(
+ (TimestampLTZNanosType(9), local.toInstant(java.time.ZoneOffset.UTC): Any, s"${ts}Z"),
+ (TimestampNTZNanosType(9), local: Any, ts)).foreach { case (nanosType, value, shown) =>
+ withTempPath { dir =>
+ val df = spark.createDataFrame(
+ spark.sparkContext.parallelize(Seq(Row(value)), numSlices = 1),
+ new StructType().add("ts", nanosType))
+ val e = intercept[SparkException] {
+ df.write.format("avro").save(dir.getCanonicalPath)
+ }
+ checkError(
+ exception = getRootCause(e).asInstanceOf[SparkArithmeticException],
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" ->
+ (s"write the timestamp value $shown as Avro epoch-nanoseconds " +
+ "(supported range: 1677-09-21T00:12:43.145224192Z to " +
+ "2262-04-11T23:47:16.854775807Z)")))
+ }
+ }
+ }
+ }
+ }
+
test("SPARK-57459: TIMESTAMP_LTZ nanos on-disk value is independent of the session time zone") {
withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
// TIMESTAMP_LTZ is an absolute instant, so the stored epoch-nanoseconds and the read-back
diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index 3ab9685acc00..502b632d0cdc 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -57,6 +57,7 @@ Spark SQL and DataFrames support the following data types:
- Note: TIMESTAMP in Spark is a user-specified alias associated with one of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations. Users can set the default timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the configuration `spark.sql.timestampType`.
- `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)`: Preview nanosecond-capable variants of `TIMESTAMP_NTZ` and `TIMESTAMP_LTZ` with fractional seconds precision `precision` in `[7, 9]`. Unparameterized `TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` remain microsecond types. In schema-driven Dataset/DataFrame conversion, Spark maps `TimestampNTZNanosType` to `java.time.LocalDateTime` and `TimestampLTZNanosType` to `java.time.Instant`; values with more sub-micro [...]
- Note: Nanosecond precision is only available through the schema-driven APIs above. Typed encoders derived from `java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a case class and `spark.createDataFrame(Seq(...))` without an explicit schema) bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` take microsecond inputs, so a nanosecond argument needs an explicit `CAST` t [...]
+ - Note: The supported range is `0001-01-01` to `9999-12-31`, as for the microsecond types. Adding an interval (`+ INTERVAL`, `timestampadd`) past the internal limit of about years -290308 to +294247 (epoch microseconds in a 64-bit long) raises `DATETIME_OVERFLOW`, and casting a string past it returns `NULL`, or raises `CAST_INVALID_INPUT` under ANSI mode. Parquet `TIMESTAMP(NANOS)` and Avro `timestamp-nanos`/`local-timestamp-nanos` store a 64-bit count of epoch nanoseconds, so they o [...]
* Interval types
- `YearMonthIntervalType(startField, endField)`: Represents a year-month interval which is made up of a contiguous subset of the following fields:
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
index 4ea71fff33c6..04ce11b9fa65 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala
@@ -1340,7 +1340,8 @@ case class UnixNanos(child: Expression)
override def inputTypes: Seq[AbstractDataType] = Seq(AnyTimestampNanoType)
// epochMicros * 1000 overflows a 64-bit BIGINT across the full [0001..9999] calendar range, so
- // the result is a lossless DECIMAL with enough precision to hold every value (~2.5e20 max).
+ // the result is a DECIMAL that holds every value in that range (~2.5e20 max). Values beyond
+ // ~+/-31,700 years from the epoch exceed 21 digits and raise NUMERIC_VALUE_OUT_OF_RANGE.
override def dataType: DataType = DecimalType(21, 0)
override def nullSafeEval(input: Any): Any = {
@@ -2610,7 +2611,8 @@ case class TimestampAddInterval(
left.dataType match {
case _: AnyTimestampNanoType =>
timestampNanosAddDayTime(
- start.asInstanceOf[TimestampNanosVal], interval.asInstanceOf[Long], zoneIdInEval)
+ start.asInstanceOf[TimestampNanosVal], interval.asInstanceOf[Long], zoneIdInEval,
+ left.dataType, right.dataType)
case _ =>
timestampAddDayTime(start.asInstanceOf[Long], interval.asInstanceOf[Long], zoneIdInEval)
}
@@ -2626,7 +2628,10 @@ case class TimestampAddInterval(
case _: DayTimeIntervalType =>
left.dataType match {
case _: AnyTimestampNanoType =>
- defineCodeGen(ctx, ev, (sd, dt) => s"""$dtu.timestampNanosAddDayTime($sd, $dt, $zid)""")
+ val st = ctx.addReferenceObj("startType", left.dataType, classOf[DataType].getName)
+ val it = ctx.addReferenceObj("intervalType", right.dataType, classOf[DataType].getName)
+ defineCodeGen(ctx, ev, (sd, dt) =>
+ s"""$dtu.timestampNanosAddDayTime($sd, $dt, $zid, $st, $it)""")
case _ =>
defineCodeGen(ctx, ev, (sd, dt) => s"""$dtu.timestampAddDayTime($sd, $dt, $zid)""")
}
@@ -2997,7 +3002,8 @@ case class TimestampAddYMInterval(
override def nullSafeEval(start: Any, months: Any): Any = left.dataType match {
case _: AnyTimestampNanoType =>
timestampNanosAddMonths(
- start.asInstanceOf[TimestampNanosVal], months.asInstanceOf[Int], zoneIdInEval)
+ start.asInstanceOf[TimestampNanosVal], months.asInstanceOf[Int], zoneIdInEval,
+ left.dataType, right.dataType)
case _ =>
timestampAddMonths(start.asInstanceOf[Long], months.asInstanceOf[Int], zoneIdInEval)
}
@@ -3007,8 +3013,10 @@ case class TimestampAddYMInterval(
val dtu = DateTimeUtils.getClass.getName.stripSuffix("$")
left.dataType match {
case _: AnyTimestampNanoType =>
+ val st = ctx.addReferenceObj("startType", left.dataType, classOf[DataType].getName)
+ val it = ctx.addReferenceObj("intervalType", right.dataType, classOf[DataType].getName)
defineCodeGen(ctx, ev, (sd, months) =>
- s"""$dtu.timestampNanosAddMonths($sd, $months, $zid)""")
+ s"""$dtu.timestampNanosAddMonths($sd, $months, $zid, $st, $it)""")
case _ =>
defineCodeGen(ctx, ev, (micros, months) =>
s"""$dtu.timestampAddMonths($micros, $months, $zid)""")
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
index f67d43c6502d..a9a89780bec7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala
@@ -28,7 +28,7 @@ import scala.util.control.NonFatal
import org.apache.spark.{QueryContext, SparkException, SparkIllegalArgumentException}
import org.apache.spark.sql.catalyst.util.DateTimeConstants._
import org.apache.spark.sql.errors.QueryExecutionErrors
-import org.apache.spark.sql.types.{Decimal, DoubleExactNumeric, TimestampNTZType, TimestampType}
+import org.apache.spark.sql.types.{DataType, Decimal, DoubleExactNumeric, TimestampNTZType, TimestampType}
import org.apache.spark.unsafe.types.{CalendarInterval, TimestampNanosVal, UTF8String}
/**
@@ -285,13 +285,22 @@ object DateTimeUtils extends SparkDateTimeUtils {
/**
* Adds a year-month interval expressed in months to a nanosecond-precision timestamp value while
- * preserving the `nanosWithinMicro` remainder.
+ * preserving the `nanosWithinMicro` remainder. Throws `DATETIME_OVERFLOW` if the result does not
+ * fit in the epoch-microseconds range; `startType` and `intervalType` render the operands in it.
*/
def timestampNanosAddMonths(
start: TimestampNanosVal,
months: Int,
- zoneId: ZoneId): TimestampNanosVal = {
- val epochMicros = timestampAddMonths(start.epochMicros, months, zoneId)
+ zoneId: ZoneId,
+ startType: DataType,
+ intervalType: DataType): TimestampNanosVal = {
+ val epochMicros = try {
+ timestampAddMonths(start.epochMicros, months, zoneId)
+ } catch {
+ case _: ArithmeticException | _: DateTimeException =>
+ throw QueryExecutionErrors.timestampNanosAddIntervalOverflowError(
+ start, startType, months, intervalType)
+ }
TimestampNanosVal.fromParts(epochMicros, start.nanosWithinMicro)
}
@@ -319,13 +328,22 @@ object DateTimeUtils extends SparkDateTimeUtils {
/**
* Adds a day-time interval to a nanosecond-precision timestamp value while preserving
- * the `nanosWithinMicro` remainder.
+ * the `nanosWithinMicro` remainder. Throws `DATETIME_OVERFLOW` if the result does not fit in the
+ * epoch-microseconds range; `startType` and `intervalType` render the operands in it.
*/
def timestampNanosAddDayTime(
start: TimestampNanosVal,
dayTime: Long,
- zoneId: ZoneId): TimestampNanosVal = {
- val epochMicros = timestampAddDayTime(start.epochMicros, dayTime, zoneId)
+ zoneId: ZoneId,
+ startType: DataType,
+ intervalType: DataType): TimestampNanosVal = {
+ val epochMicros = try {
+ timestampAddDayTime(start.epochMicros, dayTime, zoneId)
+ } catch {
+ case _: ArithmeticException | _: DateTimeException =>
+ throw QueryExecutionErrors.timestampNanosAddIntervalOverflowError(
+ start, startType, dayTime, intervalType)
+ }
TimestampNanosVal.fromParts(epochMicros, start.nanosWithinMicro)
}
@@ -338,9 +356,16 @@ object DateTimeUtils extends SparkDateTimeUtils {
* [[org.apache.spark.sql.errors.QueryExecutionErrors.timestampNanosEpochNanosOverflowError]]).
*/
def timestampNanosToEpochNanos(value: TimestampNanosVal): Long = {
- Math.addExact(
- Math.multiplyExact(value.epochMicros, NANOS_PER_MICROS),
- value.nanosWithinMicro.toLong)
+ if (value.epochMicros < 0) {
+ // Pack from `epochMicros + 1`: `epochMicros * 1000` alone overflows at the int64 minimum.
+ Math.addExact(
+ Math.multiplyExact(value.epochMicros + 1, NANOS_PER_MICROS),
+ value.nanosWithinMicro.toLong - NANOS_PER_MICROS)
+ } else {
+ Math.addExact(
+ Math.multiplyExact(value.epochMicros, NANOS_PER_MICROS),
+ value.nanosWithinMicro.toLong)
+ }
}
/**
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 9d4aadab911c..01727a9b84b7 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -2809,7 +2809,10 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
}
def timestampNanosEpochNanosOverflowError(
- value: TimestampNanosVal, isNtz: Boolean, sink: String): SparkArithmeticException = {
+ value: TimestampNanosVal,
+ isNtz: Boolean,
+ sink: String,
+ minInstant: String = "1677-09-21T00:12:43.145224192Z"): SparkArithmeticException = {
// Render TIMESTAMP_NTZ values without a zone (LocalDateTime, no trailing `Z`); TIMESTAMP_LTZ
// values are absolute instants and render as UTC with a trailing `Z`.
val rendered =
@@ -2820,7 +2823,21 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase with ExecutionE
messageParameters = Map(
"operation" -> (s"write the timestamp value $rendered as $sink " +
"epoch-nanoseconds " +
- "(supported range: 1677-09-21T00:12:43.145224192Z to 2262-04-11T23:47:16.854775807Z)")),
+ s"(supported range: $minInstant to 2262-04-11T23:47:16.854775807Z)")),
+ context = Array.empty,
+ summary = "")
+ }
+
+ def timestampNanosAddIntervalOverflowError(
+ start: TimestampNanosVal,
+ startType: DataType,
+ interval: Any,
+ intervalType: DataType): SparkArithmeticException = {
+ new SparkArithmeticException(
+ errorClass = "DATETIME_OVERFLOW",
+ messageParameters = Map(
+ "operation" ->
+ s"add ${toSQLValue(interval, intervalType)} to ${toSQLValue(start, startType)}"),
context = Array.empty,
summary = "")
}
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
index aaee630bfe51..3dcc339d49f1 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
@@ -52,6 +52,23 @@ object ArrowWriter {
new ArrowWriter(root, children.toArray)
}
+ /**
+ * Packs a nanosecond timestamp into Arrow `timestamp[ns]` epoch-nanoseconds, raising
+ * `DATETIME_OVERFLOW` outside the supported range. Unlike Parquet and Avro, the int64 minimum is
+ * out of range: pandas/numpy reserve it for `NaT`, so a non-null value would read back as null.
+ */
+ private[arrow] def timestampNanosToArrowEpochNanos(v: TimestampNanosVal, isNtz: Boolean): Long = {
+ def overflow = QueryExecutionErrors.timestampNanosEpochNanosOverflowError(
+ v, isNtz, sink = "Arrow INT64", minInstant = "1677-09-21T00:12:43.145224193Z")
+ val nanos = try {
+ DateTimeUtils.timestampNanosToEpochNanos(v)
+ } catch {
+ case _: ArithmeticException => throw overflow
+ }
+ if (nanos == Long.MinValue) throw overflow
+ nanos
+ }
+
private[sql] def createFieldWriter(vector: ValueVector): ArrowFieldWriter = {
val field = vector.getField()
val dt = ArrowUtils.fromArrowField(field)
@@ -425,15 +442,8 @@ private[sql] class TimestampNTZNanosWriter(
}
override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
- val v = input.getTimestampNTZNanos(ordinal)
- val nanos = try {
- DateTimeUtils.timestampNanosToEpochNanos(v)
- } catch {
- case _: ArithmeticException =>
- throw QueryExecutionErrors.timestampNanosEpochNanosOverflowError(
- v, isNtz = true, sink = "Arrow INT64")
- }
- valueVector.setSafe(count, nanos)
+ valueVector.setSafe(count, ArrowWriter.timestampNanosToArrowEpochNanos(
+ input.getTimestampNTZNanos(ordinal), isNtz = true))
}
}
@@ -445,15 +455,8 @@ private[sql] class TimestampLTZNanosWriter(
}
override def setValue(input: SpecializedGetters, ordinal: Int): Unit = {
- val v = input.getTimestampLTZNanos(ordinal)
- val nanos = try {
- DateTimeUtils.timestampNanosToEpochNanos(v)
- } catch {
- case _: ArithmeticException =>
- throw QueryExecutionErrors.timestampNanosEpochNanosOverflowError(
- v, isNtz = false, sink = "Arrow INT64")
- }
- valueVector.setSafe(count, nanos)
+ valueVector.setSafe(count, ArrowWriter.timestampNanosToArrowEpochNanos(
+ input.getTimestampLTZNanos(ordinal), isNtz = false))
}
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
index 674a689c1ec8..a99d1f5e4047 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastSuiteBase.scala
@@ -1728,6 +1728,32 @@ abstract class CastSuiteBase extends SparkFunSuite with ExpressionEvalHelper {
}
}
+ test("SPARK-57838: cast nanosecond timestamps at the range boundaries") {
+ // The calendar range, the int64 epoch-nanosecond limits of Parquet/Avro and the
+ // epoch-microsecond (Long) limits. Years beyond 9999 are accepted, as for the micro types.
+ Seq(
+ "0001-01-01 00:00:00" -> nanosVal(-62135596800000000L, 0),
+ "9999-12-31 23:59:59.999999999" -> nanosVal(253402300799999999L, 999),
+ "+10000-01-01 00:00:00.000000001" -> nanosVal(253402300800000000L, 1),
+ "1677-09-21 00:12:43.145224192" -> nanosVal(-9223372036854776L, 192),
+ "2262-04-11 23:47:16.854775807" -> nanosVal(9223372036854775L, 807),
+ "-290308-12-21 19:59:05.224192" -> nanosVal(Long.MinValue, 0),
+ "+294247-01-10 04:00:54.775807999" -> nanosVal(Long.MaxValue, 999)).foreach {
+ case (str, v) =>
+ checkEvaluation(cast(Literal(str), TimestampNTZNanosType(9)), v)
+ checkEvaluation(cast(Literal(s"${str}Z"), TimestampLTZNanosType(9), UTC_OPT), v)
+ checkEvaluation(cast(Literal.create(v, TimestampNTZNanosType(9)), StringType), str)
+ checkEvaluation(
+ cast(Literal.create(v, TimestampLTZNanosType(9)), StringType, UTC_OPT), str)
+ // To and from the micro type keeps the epoch-microseconds part.
+ checkEvaluation(
+ cast(Literal.create(v, TimestampNTZNanosType(9)), TimestampNTZType), v.epochMicros)
+ checkEvaluation(
+ cast(Literal.create(v.epochMicros, TimestampNTZType), TimestampNTZNanosType(9)),
+ nanosVal(v.epochMicros, 0))
+ }
+ }
+
test("SPARK-57323: cast between date and timestamp_ntz with nanosecond precision") {
// NTZ casts use a fixed UTC wall-clock grid, independent of the session time zone.
val date = LocalDate.of(2020, 1, 1)
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
index 715144b13ad3..2ae5825f5ff2 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOffSuite.scala
@@ -68,6 +68,15 @@ class CastWithAnsiOffSuite extends CastSuiteBase {
}
}
+ test("SPARK-57838: legacy mode cast string beyond the epoch-microsecond range to nanos") {
+ // Same NULL as the micro types: the value does not fit the Long epoch-microseconds.
+ Seq("+294247-01-10 04:00:54.775808", "-290308-12-21 19:59:05.224191999").foreach { str =>
+ checkEvaluation(cast(Literal(str), TimestampNTZType), null)
+ checkEvaluation(cast(Literal(str), TimestampNTZNanosType(9)), null)
+ checkEvaluation(cast(Literal(s"${str}Z"), TimestampLTZNanosType(9), UTC_OPT), null)
+ }
+ }
+
test("cast from int #2") {
checkEvaluation(cast(cast(1000, TimestampType), LongType), 1000.toLong)
checkEvaluation(cast(cast(-1200, TimestampType), LongType), -1200.toLong)
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
index 0c3bcede5e07..a595fd8e0a95 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala
@@ -862,6 +862,19 @@ class CastWithAnsiOnSuite extends CastSuiteBase with QueryErrorsBase {
}
}
+ test("SPARK-57838: ANSI mode cast string beyond the epoch-microsecond range to nanos") {
+ // Same error as the micro types: the value does not fit the Long epoch-microseconds.
+ Seq("+294247-01-10 04:00:54.775808", "-290308-12-21 19:59:05.224191999").foreach { str =>
+ checkExceptionInExpression[DateTimeException](
+ cast(Literal(str), TimestampNTZType), castErrMsg(str, TimestampNTZType))
+ checkExceptionInExpression[DateTimeException](
+ cast(Literal(str), TimestampNTZNanosType(9)), castErrMsg(str, TimestampNTZNanosType(9)))
+ checkExceptionInExpression[DateTimeException](
+ cast(Literal(s"${str}Z"), TimestampLTZNanosType(9), DateTimeTestUtils.UTC_OPT),
+ castErrMsg(s"${str}Z", TimestampLTZNanosType(9)))
+ }
+ }
+
test("ANSI mode: cast string to date with parse error") {
DateTimeTestUtils.outstandingZoneIds.foreach { zid =>
def checkCastWithParseError(str: String): Unit = {
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
index de3773f14500..cc0da8f1c5cf 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DateExpressionsSuite.scala
@@ -37,6 +37,7 @@ import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils, Timesta
import org.apache.spark.sql.catalyst.util.DateTimeConstants._
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils.{getZoneId, TimeZoneUTC}
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.nanosVal
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.TimestampTypes
import org.apache.spark.sql.types._
@@ -2465,6 +2466,98 @@ class DateExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}
+ test("SPARK-57838: nanos timestamp arithmetic at the range boundaries") {
+ def utc(s: String): TimestampNanosVal =
+ DateTimeUtils.localDateTimeToTimestampNanos(LocalDateTime.parse(s), precision = 9)
+ val maxCalendar = utc("9999-12-31T23:59:59.999999999")
+ val max = nanosVal(Long.MaxValue, 999)
+ val min = nanosVal(Long.MinValue, 0)
+ val int64Max = DateTimeUtils.epochNanosToTimestampNanos(Long.MaxValue, 9)
+ val int64Min = DateTimeUtils.epochNanosToTimestampNanos(Long.MinValue, 9)
+
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ Seq(
+ TimestampNTZNanosType(9) -> "TIMESTAMP_NTZ",
+ TimestampLTZNanosType(9) -> "TIMESTAMP_LTZ").foreach { case (t, typeName) =>
+ def lit(v: TimestampNanosVal): Literal = Literal.create(v, t)
+ // Crossing 9999-12-31 is not an error, as for the microsecond types.
+ checkEvaluation(
+ TimestampAddInterval(lit(maxCalendar), Literal(Duration.ofDays(1)), Some("UTC")),
+ utc("+10000-01-01T23:59:59.999999999"))
+ checkEvaluation(
+ TimestampAddYMInterval(lit(maxCalendar), Literal(Period.ofYears(1)), Some("UTC")),
+ utc("+10000-12-31T23:59:59.999999999"))
+ checkEvaluation(
+ TimestampAdd("DAY", Literal(1L), lit(maxCalendar), Some("UTC")),
+ utc("+10000-01-01T23:59:59.999999999"))
+ // The int64 epoch-nanosecond limits are ordinary values: no wrap-around past them.
+ checkEvaluation(
+ TimestampAdd("NANOSECOND", Literal(1L), lit(int64Max), Some("UTC")),
+ nanosVal(int64Max.epochMicros, int64Max.nanosWithinMicro + 1))
+ checkEvaluation(
+ TimestampAdd("NANOSECOND", Literal(-1L), lit(int64Min), Some("UTC")),
+ nanosVal(int64Min.epochMicros, int64Min.nanosWithinMicro - 1))
+
+ // Past the epoch-microsecond limits, interval arithmetic raises DATETIME_OVERFLOW, with
+ // the operands rendered as SQL literals of their own types.
+ val maxSql = s"$typeName '+294247-01-10 04:00:54.775807999'"
+ checkErrorInExpression[SparkArithmeticException](
+ TimestampAddInterval(lit(max), Literal(Duration.ofDays(1)), Some("UTC")),
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" -> s"add INTERVAL '1 00:00:00' DAY TO SECOND to $maxSql"))
+ checkErrorInExpression[SparkArithmeticException](
+ TimestampAddInterval(
+ lit(max), Literal(MICROS_PER_HOUR, DayTimeIntervalType(DayTimeIntervalType.HOUR)),
+ Some("UTC")),
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" -> s"add INTERVAL '01' HOUR to $maxSql"))
+ checkErrorInExpression[SparkArithmeticException](
+ TimestampAddInterval(lit(min), UnaryMinus(Literal(Duration.ofDays(1))), Some("UTC")),
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" -> (s"add INTERVAL '-1 00:00:00' DAY TO SECOND to " +
+ s"$typeName '-290308-12-21 19:59:05.224192000'")))
+ checkErrorInExpression[SparkArithmeticException](
+ TimestampAddYMInterval(
+ lit(max), Literal(1, YearMonthIntervalType(YearMonthIntervalType.MONTH)), Some("UTC")),
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" -> s"add INTERVAL '1' MONTH to $maxSql"))
+ }
+ }
+ }
+
+ test("SPARK-57838: unix_nanos at the range boundaries") {
+ def unixNanos(v: TimestampNanosVal): UnixNanos =
+ UnixNanos(Literal.create(v, TimestampNTZNanosType(9)))
+ def decimal(n: BigInt): Decimal = Decimal(BigDecimal(n), 21, 0)
+ // The int64 epoch-nanosecond limits and the calendar range map to exact values.
+ checkEvaluation(
+ unixNanos(DateTimeUtils.epochNanosToTimestampNanos(Long.MaxValue, 9)),
+ decimal(Long.MaxValue))
+ checkEvaluation(
+ unixNanos(DateTimeUtils.epochNanosToTimestampNanos(Long.MinValue, 9)),
+ decimal(Long.MinValue))
+ checkEvaluation(
+ unixNanos(DateTimeUtils.localDateTimeToTimestampNanos(
+ LocalDateTime.parse("0001-01-01T00:00:00"), precision = 9)),
+ decimal(BigInt("-62135596800000000000")))
+ // DECIMAL(21, 0) holds +/-(10^21 - 1) nanoseconds (about +/-31,700 years from the epoch);
+ // timestamps further out raise NUMERIC_VALUE_OUT_OF_RANGE instead of wrapping.
+ val maxDecimalMicros = (BigInt(10).pow(21) - 1) / 1000
+ checkEvaluation(
+ unixNanos(TimestampNanosVal.fromParts(maxDecimalMicros.toLong, 999.toShort)),
+ decimal(BigInt(10).pow(21) - 1))
+ checkEvaluation(
+ unixNanos(TimestampNanosVal.fromParts(-maxDecimalMicros.toLong - 1, 1.toShort)),
+ decimal(-(BigInt(10).pow(21) - 1)))
+ Seq(
+ TimestampNanosVal.fromParts(maxDecimalMicros.toLong + 1, 0.toShort),
+ TimestampNanosVal.fromParts(Long.MaxValue, 999.toShort),
+ TimestampNanosVal.fromParts(Long.MinValue, 0.toShort)).foreach { v =>
+ checkExceptionInExpression[SparkArithmeticException](
+ unixNanos(v), "NUMERIC_VALUE_OUT_OF_RANGE")
+ }
+ }
+
test("SPARK-57832: subtract nanosecond-precision timestamps") {
// The difference between two nanosecond timestamps is reported on the microsecond grid: only
// each operand's epochMicros participates, so the sub-microsecond remainder is truncated. The
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
index 5b71273e92e5..1d39e88d9d54 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/DateTimeUtilsSuite.scala
@@ -35,8 +35,8 @@ import org.apache.spark.sql.catalyst.util.RebaseDateTime.rebaseJulianToGregorian
import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.nanosVal
import org.apache.spark.sql.errors.DataTypeErrors.toSQLConf
import org.apache.spark.sql.internal.SqlApiConf
-import org.apache.spark.sql.types.{Decimal, TimeType}
-import org.apache.spark.sql.types.DayTimeIntervalType.{HOUR, SECOND}
+import org.apache.spark.sql.types.{DayTimeIntervalType, Decimal, TimestampLTZNanosType, TimestampNTZNanosType, TimeType, YearMonthIntervalType}
+import org.apache.spark.sql.types.DayTimeIntervalType.{DAY, HOUR, SECOND}
import org.apache.spark.unsafe.types.{CalendarInterval, TimestampNanosVal, UTF8String}
class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
@@ -1158,12 +1158,15 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
test("SPARK-57501: timestamp nanos add day-time interval preserves nanosWithinMicro") {
def nanos(epochMicros: Long, nanosWithinMicro: Int): TimestampNanosVal =
TimestampNanosVal.fromParts(epochMicros, nanosWithinMicro.toShort)
+ // The operand types only render the overflow error.
+ val ltz = TimestampLTZNanosType(9)
+ val dayTimeType = DayTimeIntervalType()
// The epoch-micros part follows the micro `timestampAddDayTime` (including the transition from
// Pacific Standard to Pacific Daylight Time) while the sub-microsecond remainder is carried
// through unchanged.
assert(timestampNanosAddDayTime(
- nanos(date(2019, 3, 9, 12, 0, 0, 123000, LA), 789), MICROS_PER_DAY, LA) ===
+ nanos(date(2019, 3, 9, 12, 0, 0, 123000, LA), 789), MICROS_PER_DAY, LA, ltz, dayTimeType) ===
nanos(date(2019, 3, 10, 12, 0, 0, 123000, LA), 789))
outstandingZoneIds.foreach { zid =>
@@ -1171,20 +1174,23 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
Seq(0, 1, 999).foreach { rem =>
// Zero interval is a no-op on both the epoch-micros and the remainder.
assert(timestampNanosAddDayTime(
- nanos(date(2021, 3, 18, 19, 44, 1, 100000, zid), rem), 0, zid) ===
+ nanos(date(2021, 3, 18, 19, 44, 1, 100000, zid), rem), 0, zid, ltz, dayTimeType) ===
nanos(date(2021, 3, 18, 19, 44, 1, 100000, zid), rem))
// Subtracting whole days shifts only the epoch-micros part.
assert(timestampNanosAddDayTime(
- nanos(date(2021, 1, 19, 0, 0, 0, 0, zid), rem), -18 * MICROS_PER_DAY, zid) ===
+ nanos(date(2021, 1, 19, 0, 0, 0, 0, zid), rem), -18 * MICROS_PER_DAY, zid,
+ ltz, dayTimeType) ===
nanos(date(2021, 1, 1, 0, 0, 0, 0, zid), rem))
// A +1 microsecond carry from the day-time interval only moves epochMicros
// (123456 -> 123457), never the remainder.
assert(timestampNanosAddDayTime(
- nanos(date(2019, 5, 9, 12, 0, 0, 123456, zid), rem), 2 * MICROS_PER_DAY + 1, zid) ===
+ nanos(date(2019, 5, 9, 12, 0, 0, 123456, zid), rem), 2 * MICROS_PER_DAY + 1, zid,
+ ltz, dayTimeType) ===
nanos(date(2019, 5, 11, 12, 0, 0, 123457, zid), rem))
// Pre-epoch (negative epochMicros) value.
assert(timestampNanosAddDayTime(
- nanos(date(1960, 1, 2, 3, 4, 5, 123456, zid), rem), MICROS_PER_DAY, zid) ===
+ nanos(date(1960, 1, 2, 3, 4, 5, 123456, zid), rem), MICROS_PER_DAY, zid,
+ ltz, dayTimeType) ===
nanos(date(1960, 1, 3, 3, 4, 5, 123456, zid), rem))
}
}
@@ -1194,7 +1200,7 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
outstandingZoneIds.foreach { zid =>
val start = nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), 789)
val dayTime = 3 * MICROS_PER_HOUR + 7
- val result = timestampNanosAddDayTime(start, dayTime, zid)
+ val result = timestampNanosAddDayTime(start, dayTime, zid, ltz, dayTimeType)
assert(result.epochMicros === timestampAddDayTime(start.epochMicros, dayTime, zid))
assert(result.nanosWithinMicro === start.nanosWithinMicro)
}
@@ -1203,11 +1209,14 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
test("SPARK-57825: timestamp nanos add year-month interval preserves nanosWithinMicro") {
def nanos(epochMicros: Long, nanosWithinMicro: Int): TimestampNanosVal =
TimestampNanosVal.fromParts(epochMicros, nanosWithinMicro.toShort)
+ // The operand types only render the overflow error.
+ val ltz = TimestampLTZNanosType(9)
+ val yearMonthType = YearMonthIntervalType()
// The epoch-micros part follows the micro `timestampAddMonths` (including the Jan-31 -> Feb-29
// day clamp in a leap year) while the sub-microsecond remainder is carried through unchanged.
assert(timestampNanosAddMonths(
- nanos(date(2020, 1, 31, 12, 0, 0, 123000, LA), 789), 1, LA) ===
+ nanos(date(2020, 1, 31, 12, 0, 0, 123000, LA), 789), 1, LA, ltz, yearMonthType) ===
nanos(date(2020, 2, 29, 12, 0, 0, 123000, LA), 789))
outstandingZoneIds.foreach { zid =>
@@ -1215,19 +1224,19 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
Seq(0, 1, 999).foreach { rem =>
// Zero interval is a no-op on both the epoch-micros and the remainder.
assert(timestampNanosAddMonths(
- nanos(date(2021, 3, 18, 19, 44, 1, 123456, zid), rem), 0, zid) ===
+ nanos(date(2021, 3, 18, 19, 44, 1, 123456, zid), rem), 0, zid, ltz, yearMonthType) ===
nanos(date(2021, 3, 18, 19, 44, 1, 123456, zid), rem))
// Adding whole years/months shifts only the month field, never the fraction.
assert(timestampNanosAddMonths(
- nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), rem), 14, zid) ===
+ nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), rem), 14, zid, ltz, yearMonthType) ===
nanos(date(2021, 3, 2, 3, 4, 5, 123456, zid), rem))
// Subtracting months is symmetric.
assert(timestampNanosAddMonths(
- nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), rem), -1, zid) ===
+ nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), rem), -1, zid, ltz, yearMonthType) ===
nanos(date(2019, 12, 2, 3, 4, 5, 123456, zid), rem))
// Pre-epoch (negative epochMicros) value.
assert(timestampNanosAddMonths(
- nanos(date(1960, 1, 2, 3, 4, 5, 123456, zid), rem), 1, zid) ===
+ nanos(date(1960, 1, 2, 3, 4, 5, 123456, zid), rem), 1, zid, ltz, yearMonthType) ===
nanos(date(1960, 2, 2, 3, 4, 5, 123456, zid), rem))
}
}
@@ -1237,12 +1246,62 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
outstandingZoneIds.foreach { zid =>
val start = nanos(date(2020, 1, 2, 3, 4, 5, 123456, zid), 789)
val months = 15
- val result = timestampNanosAddMonths(start, months, zid)
+ val result = timestampNanosAddMonths(start, months, zid, ltz, yearMonthType)
assert(result.epochMicros === timestampAddMonths(start.epochMicros, months, zid))
assert(result.nanosWithinMicro === start.nanosWithinMicro)
}
}
+ test("SPARK-57838: timestamp nanos add interval at the range boundaries") {
+ val ntz = TimestampNTZNanosType(9)
+ val dayTimeType = DayTimeIntervalType()
+ val yearMonthType = YearMonthIntervalType()
+ val max = nanosVal(Long.MaxValue, 999)
+ val min = nanosVal(Long.MinValue, 0)
+
+ // Crossing 9999-12-31 or 0001-01-01 is not an error, as for the microsecond types.
+ val maxCalendar = nanosVal(date(9999, 12, 31, 23, 59, 59, 999999, UTC), 999)
+ assert(timestampNanosAddDayTime(maxCalendar, MICROS_PER_DAY, UTC, ntz, dayTimeType) ===
+ nanosVal(date(10000, 1, 1, 23, 59, 59, 999999, UTC), 999))
+ assert(timestampNanosAddMonths(maxCalendar, 1, UTC, ntz, yearMonthType) ===
+ nanosVal(date(10000, 1, 31, 23, 59, 59, 999999, UTC), 999))
+ assert(timestampNanosAddDayTime(
+ nanosVal(date(1, 1, 1, 0, 0, 0, 0, UTC), 1), -1, UTC, ntz, dayTimeType) ===
+ nanosVal(date(0, 12, 31, 23, 59, 59, 999999, UTC), 1))
+ // The epoch-microsecond limits themselves are reachable.
+ assert(timestampNanosAddDayTime(
+ nanosVal(Long.MaxValue - MICROS_PER_DAY, 999), MICROS_PER_DAY, UTC, ntz, dayTimeType) === max)
+ assert(timestampNanosAddDayTime(
+ nanosVal(Long.MinValue + 1, 0), -1, UTC, ntz, dayTimeType) === min)
+
+ // Past them the addition raises DATETIME_OVERFLOW instead of a raw ArithmeticException, with
+ // the operands rendered as SQL literals of their own types.
+ withSQLConf(SqlApiConf.SESSION_LOCAL_TIMEZONE_KEY -> LA.getId) {
+ Seq(
+ (() => timestampNanosAddDayTime(max, 1, UTC, ntz, dayTimeType),
+ "add INTERVAL '0 00:00:00.000001' DAY TO SECOND to " +
+ "TIMESTAMP_NTZ '+294247-01-10 04:00:54.775807999'"),
+ (() => timestampNanosAddDayTime(
+ min, -MICROS_PER_DAY, UTC, TimestampNTZNanosType(7), DayTimeIntervalType(DAY)),
+ "add INTERVAL '-1' DAY to TIMESTAMP_NTZ '-290308-12-21 19:59:05.2241920'"),
+ (() => timestampNanosAddMonths(
+ max, 1, UTC, ntz, YearMonthIntervalType(YearMonthIntervalType.MONTH)),
+ "add INTERVAL '1' MONTH to TIMESTAMP_NTZ '+294247-01-10 04:00:54.775807999'"),
+ (() => timestampNanosAddMonths(min, -12, UTC, ntz, yearMonthType),
+ "add INTERVAL '-1-0' YEAR TO MONTH to TIMESTAMP_NTZ '-290308-12-21 19:59:05.224192000'"),
+ // TIMESTAMP_LTZ renders in the session time zone.
+ (() => timestampNanosAddDayTime(
+ max, MICROS_PER_DAY, LA, TimestampLTZNanosType(9), DayTimeIntervalType(HOUR)),
+ "add INTERVAL '24' HOUR to TIMESTAMP_LTZ '+294247-01-09 20:00:54.775807999'")).foreach {
+ case (f, operation) =>
+ checkError(
+ exception = intercept[SparkArithmeticException](f()),
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" -> operation))
+ }
+ }
+ }
+
test("SPARK-57159: timestampNanosToEpochNanos packs into int64 epoch-nanoseconds") {
def nanos(epochMicros: Long, nanosWithinMicro: Int): TimestampNanosVal =
TimestampNanosVal.fromParts(epochMicros, nanosWithinMicro.toShort)
@@ -1265,14 +1324,28 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
assert(Math.floorMod(packed, NANOS_PER_MICROS) === v.nanosWithinMicro.toLong)
}
- // The maximum representable instant packs exactly to Long.MaxValue.
+ // The maximum and minimum representable instants pack exactly to Long.MaxValue and
+ // Long.MinValue (2262-04-11T23:47:16.854775807Z and 1677-09-21T00:12:43.145224192Z).
assert(timestampNanosToEpochNanos(
nanos(Long.MaxValue / NANOS_PER_MICROS, (Long.MaxValue % NANOS_PER_MICROS).toInt)) ===
Long.MaxValue)
+ assert(timestampNanosToEpochNanos(nanos(Long.MinValue / NANOS_PER_MICROS - 1, 192)) ===
+ Long.MinValue)
+ assert(timestampNanosToEpochNanos(nanos(Long.MinValue / NANOS_PER_MICROS - 1, 999)) ===
+ Long.MinValue + 807)
- // Out-of-range values raise ArithmeticException (callers translate it into DATETIME_OVERFLOW).
- intercept[ArithmeticException] {
- timestampNanosToEpochNanos(nanos(Long.MaxValue / NANOS_PER_MICROS + 1, 0))
+ // Out-of-range values, including one nanosecond beyond either limit, raise
+ // ArithmeticException (callers translate it into DATETIME_OVERFLOW).
+ Seq(
+ nanos(Long.MaxValue / NANOS_PER_MICROS, 808),
+ nanos(Long.MaxValue / NANOS_PER_MICROS + 1, 0),
+ nanos(Long.MinValue / NANOS_PER_MICROS - 1, 191),
+ nanos(Long.MinValue / NANOS_PER_MICROS - 2, 999),
+ nanos(Long.MaxValue, 999),
+ nanos(Long.MinValue, 0)).foreach { v =>
+ intercept[ArithmeticException] {
+ timestampNanosToEpochNanos(v)
+ }
}
}
@@ -1304,12 +1377,10 @@ class DateTimeUtilsSuite extends SparkFunSuite with Matchers with SQLHelper {
assert(epochNanosToTimestampNanos(Long.MaxValue, 9) === nanos(9223372036854775L, 807))
assert(epochNanosToTimestampNanos(Long.MinValue, 9) === nanos(-9223372036854776L, 192))
- // Round-trips with timestampNanosToEpochNanos at full precision. Long.MinValue is excluded: its
- // decode (-9223372036854776, 192) re-encodes through an intermediate epochMicros * 1000 that
- // overflows Long, an existing limitation of the multiplyExact-based pack path.
+ // Round-trips with timestampNanosToEpochNanos at full precision, including both int64 extremes.
Seq(
0L, 999L, 1234567L * NANOS_PER_MICROS + 7L, -1234567L * NANOS_PER_MICROS + 13L,
- Long.MaxValue).foreach { epochNanos =>
+ Long.MaxValue, Long.MinValue, Long.MinValue + 807).foreach { epochNanos =>
assert(timestampNanosToEpochNanos(epochNanosToTimestampNanos(epochNanos, 9)) === epochNanos)
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampNanosParseSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampNanosParseSuite.scala
index 3a4d758da892..21111311c143 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampNanosParseSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/TimestampNanosParseSuite.scala
@@ -22,6 +22,7 @@ import java.time.{ZoneId, ZoneOffset}
import org.apache.spark.{SparkDateTimeException, SparkException, SparkFunSuite}
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils._
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.nanosVal
import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
/**
@@ -146,6 +147,55 @@ class TimestampNanosParseSuite extends SparkFunSuite {
date(9999, 12, 31, 23, 59, 59, 999999, ZoneOffset.UTC), 999.toShort))
}
+ test("SPARK-57838: parse and format at the calendar and int64 boundaries") {
+ val formatter = TimestampFormatter.getFractionFormatter(ZoneOffset.UTC)
+ val minMicros = nanosVal(Long.MinValue, 0)
+ val maxMicros = nanosVal(Long.MaxValue, 999)
+ Seq(
+ // Calendar range 0001-01-01 .. 9999-12-31.
+ "0001-01-01 00:00:00" -> nanosVal(date(1, 1, 1, 0, 0, 0, 0, ZoneOffset.UTC), 0),
+ "9999-12-31 23:59:59.999999999" ->
+ nanosVal(date(9999, 12, 31, 23, 59, 59, 999999, ZoneOffset.UTC), 999),
+ // The int64 epoch-nanosecond limits of Parquet/Avro, and one nanosecond beyond them.
+ "1677-09-21 00:12:43.145224192" -> epochNanosToTimestampNanos(Long.MinValue, 9),
+ "1677-09-21 00:12:43.145224191" -> nanosVal(Long.MinValue / 1000 - 1, 191),
+ "2262-04-11 23:47:16.854775807" -> epochNanosToTimestampNanos(Long.MaxValue, 9),
+ "2262-04-11 23:47:16.854775808" -> nanosVal(Long.MaxValue / 1000, 808),
+ // Years beyond 9999 are accepted, as for the microsecond types.
+ "+10000-01-01 00:00:00.000000001" ->
+ nanosVal(date(10000, 1, 1, 0, 0, 0, 0, ZoneOffset.UTC), 1),
+ // The epoch-microsecond (Long) limits.
+ "-290308-12-21 19:59:05.224192" -> minMicros,
+ "+294247-01-10 04:00:54.775807999" -> maxMicros).foreach { case (str, expected) =>
+ assert(ntz(str, 9).get === expected)
+ assert(ltz(s"${str}Z", 9, ZoneOffset.UTC).get === expected)
+ assert(formatter.formatWithoutTimeZoneNanos(expected, 9) === str)
+ assert(formatter.formatNanos(expected, 9) === str)
+ // The epoch-microseconds part matches the microsecond parser.
+ assert(stringToTimestampWithoutTimeZone(UTF8String.fromString(str), allowTimeZone = true)
+ === Some(expected.epochMicros))
+ }
+
+ // Past the epoch-microsecond limits parsing fails like the microsecond parser (NULL, or
+ // CAST_INVALID_INPUT under ANSI).
+ Seq("+294247-01-10 04:00:54.775808", "-290308-12-21 19:59:05.224191999").foreach { str =>
+ assert(ntz(str, 9).isEmpty)
+ assert(ltz(s"${str}Z", 9, ZoneOffset.UTC).isEmpty)
+ assert(stringToTimestampWithoutTimeZone(UTF8String.fromString(str), allowTimeZone = true)
+ .isEmpty)
+ checkError(
+ exception = intercept[SparkDateTimeException] {
+ stringToTimestampNTZNanosAnsi(UTF8String.fromString(str), 9)
+ },
+ condition = "CAST_INVALID_INPUT",
+ parameters = Map(
+ "expression" -> s"'$str'",
+ "sourceType" -> "\"STRING\"",
+ "targetType" -> "\"TIMESTAMP_NTZ(9)\"",
+ "ansiConfig" -> "\"spark.sql.ansi.enabled\""))
+ }
+ }
+
test("null input returns None") {
assert(stringToTimestampNTZNanos(null, 9).isEmpty)
assert(stringToTimestampLTZNanos(null, 9, ZoneOffset.UTC).isEmpty)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
index 6e8e2deeba67..964a5767bd8b 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowWriterSuite.scala
@@ -191,7 +191,10 @@ class ArrowWriterSuite extends SparkFunSuite {
TimestampNanosVal.fromParts(-1234567L, 13.toShort),
// large positive/negative epoch-nanoseconds within the representable range
fromEpochNanos(9000000000000000000L),
- fromEpochNanos(-9000000000000000000L))
+ fromEpochNanos(-9000000000000000000L),
+ // the Arrow limits: the int64 minimum itself is NaT in pandas (see the test below)
+ fromEpochNanos(Long.MaxValue),
+ fromEpochNanos(Long.MinValue + 1))
def check(dt: DataType, timeZoneId: String): Unit = {
val schema = new StructType().add("value", dt, nullable = true)
@@ -230,13 +233,32 @@ class ArrowWriterSuite extends SparkFunSuite {
val schema = new StructType().add("value", dt, nullable = true)
val writer = ArrowWriter.create(schema, timeZoneId)
// epochMicros past the int64 epoch-nanosecond range overflows when packed, but still
- // renders as a valid Instant/LocalDateTime in the error message.
- val tooLarge = TimestampNanosVal.fromParts(Long.MaxValue / 1000L + 1L, 0.toShort)
- val e = intercept[SparkArithmeticException] {
- writer.write(InternalRow(tooLarge))
+ // renders as a valid Instant/LocalDateTime in the error message. One nanosecond past either
+ // limit overflows too.
+ Seq(
+ TimestampNanosVal.fromParts(Long.MaxValue / 1000L + 1L, 0.toShort),
+ TimestampNanosVal.fromParts(Long.MaxValue / 1000L, 808.toShort),
+ TimestampNanosVal.fromParts(Long.MinValue / 1000L - 1L, 191.toShort)).foreach { v =>
+ val e = intercept[SparkArithmeticException] {
+ writer.write(InternalRow(v))
+ }
+ assert(e.getCondition === "DATETIME_OVERFLOW")
+ assert(e.getMessage.contains("Arrow INT64"))
+ }
+ // The int64 minimum is rejected too: pandas/numpy read it as NaT.
+ val int64Min = TimestampNanosVal.fromParts(Long.MinValue / 1000L - 1L, 192.toShort)
+ val rendered = dt match {
+ case _: TimestampNTZNanosType => "1677-09-21T00:12:43.145224192"
+ case _ => "1677-09-21T00:12:43.145224192Z"
}
- assert(e.getCondition === "DATETIME_OVERFLOW")
- assert(e.getMessage.contains("Arrow INT64"))
+ checkError(
+ exception = intercept[SparkArithmeticException] {
+ writer.write(InternalRow(int64Min))
+ },
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" ->
+ (s"write the timestamp value $rendered as Arrow INT64 epoch-nanoseconds " +
+ "(supported range: 1677-09-21T00:12:43.145224193Z to 2262-04-11T23:47:16.854775807Z)")))
writer.root.close()
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTimestampNanosSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTimestampNanosSuite.scala
index 44523253525d..982822242931 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTimestampNanosSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTimestampNanosSuite.scala
@@ -385,6 +385,69 @@ class ParquetTimestampNanosSuite extends QueryTest with ParquetTest with SharedS
}
}
+ test("SPARK-57838: Parquet INT64 nanos at the exact int64 epoch-nanos limits") {
+ withNanosEnabled {
+ withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
+ val min = "1677-09-21 00:12:43.145224192"
+ val max = "2262-04-11 23:47:16.854775807"
+ withTempPath { dir =>
+ val path = dir.getCanonicalPath
+ val df = spark.sql(
+ s"""SELECT * FROM VALUES
+ | (TIMESTAMP_NTZ '$min', TIMESTAMP_LTZ '$min'),
+ | (TIMESTAMP_NTZ '$max', TIMESTAMP_LTZ '$max')
+ | AS t(ntz, ltz)""".stripMargin)
+ df.write.parquet(path)
+ def filterAt(ts: String): String =
+ s"ntz = TIMESTAMP_NTZ '$ts' AND ltz = TIMESTAMP_LTZ '$ts'"
+ withAllParquetReaders {
+ val read = spark.read.parquet(path)
+ checkAnswer(read, df.collect().toSeq)
+ withSQLConf(SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true") {
+ Seq(min, max).foreach { ts =>
+ checkAnswer(read.where(filterAt(ts)), df.where(filterAt(ts)).collect().toSeq)
+ }
+ }
+ }
+ // The filters on the limits are pushed down and encode them exactly: with record-level
+ // filtering, Parquet alone returns exactly the matching row.
+ withSQLConf(
+ SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "true",
+ SQLConf.PARQUET_RECORD_FILTER_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "false") {
+ Seq(min, max).foreach { ts =>
+ assert(stripSparkFilter(spark.read.parquet(path).where(filterAt(ts))).count() === 1)
+ }
+ }
+ }
+ // One nanosecond past either limit fails to write.
+ Seq(
+ "1677-09-21 00:12:43.145224191" -> "1677-09-21T00:12:43.145224191",
+ "2262-04-11 23:47:16.854775808" -> "2262-04-11T23:47:16.854775808").foreach {
+ case (ts, rendered) =>
+ Seq("TIMESTAMP_NTZ" -> rendered, "TIMESTAMP_LTZ" -> s"${rendered}Z").foreach {
+ case (typeName, renderedValue) =>
+ withTempPath { dir =>
+ val e = intercept[SparkException] {
+ spark.sql(s"SELECT $typeName '$ts' AS ts").write.parquet(dir.getCanonicalPath)
+ }
+ val cause = Iterator.iterate[Throwable](e)(_.getCause).takeWhile(_ != null)
+ .collectFirst { case a: SparkArithmeticException => a }
+ assert(cause.isDefined, s"Expected DATETIME_OVERFLOW, but got: ${e.getMessage}")
+ checkError(
+ exception = cause.get,
+ condition = "DATETIME_OVERFLOW",
+ parameters = Map("operation" ->
+ (s"write the timestamp value $renderedValue as Parquet INT64 " +
+ "epoch-nanoseconds (supported range: 1677-09-21T00:12:43.145224192Z to " +
+ "2262-04-11T23:47:16.854775807Z)")))
+ }
+ }
+ }
+ }
+ }
+ }
+
test("SPARK-57102: datetime rebase configs do not affect TIMESTAMP(NANOS) reads") {
withNanosEnabled {
withSQLConf(SQLConf.SESSION_LOCAL_TIMEZONE.key -> "UTC") {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimestampNanosParquetOpsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimestampNanosParquetOpsSuite.scala
index b1c934332b68..4cb1a13ca415 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimestampNanosParquetOpsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/types/ops/TimestampNanosParquetOpsSuite.scala
@@ -272,6 +272,19 @@ class TimestampNanosParquetOpsSuite extends SparkFunSuite {
assert(ntzOps.acceptsValue(LocalDateTime.parse("2020-01-01T00:00:00")))
}
+ test("SPARK-57838: filterOps accepts exactly the INT64 epoch-nanos range") {
+ val ltzOps = TimestampNanosParquetOps.ltzFilterOps
+ val ntzOps = TimestampNanosParquetOps.ntzFilterOps
+ Seq(
+ "1677-09-21T00:12:43.145224192" -> true,
+ "2262-04-11T23:47:16.854775807" -> true,
+ "1677-09-21T00:12:43.145224191" -> false,
+ "2262-04-11T23:47:16.854775808" -> false).foreach { case (ts, inRange) =>
+ assert(ltzOps.acceptsValue(Instant.parse(s"${ts}Z")) === inRange, ts)
+ assert(ntzOps.acceptsValue(LocalDateTime.parse(ts)) === inRange, ts)
+ }
+ }
+
test("filterOps make* throws on an out-of-range value - why acceptsValue must gate every In " +
"element, not just the head") {
// The encoder is exact (Math.addExact/multiplyExact), so building a predicate directly from an
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org