You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "MaxGekk (via GitHub)" <gi...@apache.org> on 2026/05/18 08:50:18 UTC
[PR] [WIP][SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
MaxGekk opened a new pull request, #55952:
URL: https://github.com/apache/spark/pull/55952
### What changes were proposed in this pull request?
In the PR, I propose to extend the Spark SQL type system, and add new classes to Scala/Java APIs:
* TimestampNTZNanosType(p)represents the SQL data type TIMESTAMP\_NTZ(p)
* TimestampLTZNanosType(p)represents TIMESTAMP\_LTZ(p)
They align with the SQL standard’s direction for optional feature F555, “Enhanced seconds precision”: datetime types can carry fractional seconds with precision p in the SECOND field beyond the traditional six decimal places (microseconds). Here p is restricted to 7, 8, and 9, i.e. the nanosecond-capable band (up to nine fractional digits, nanoseconds in the second field).
The logical layout documented on the classes matches this precision story: epoch microseconds plus nanoseconds within that microsecond, with a default estimated width of 10 bytes for planning (8 \+ 2).
Parameterless timestamp\_ntz / timestamp\_ltz are unchanged and remain the existing microsecond-oriented types.
### Why are the changes needed?
New timestamp types are useful for Spark SQL users because they allow:
1. Represent timestamp without time zone and timestamp with local time zone with fractional-second precision 7–9, in line with SQL optional feature F555 (Enhanced seconds precision).
2. Describe schemas from other systems that already use nanosecond-capable timestamps, without overloading microsecond timestamp\_ntz / timestamp\_ltz types.
3. Migrate SQL and metadata that distinguish NTZ and LTZ at sub-microsecond precision toward Spark in small, reviewable steps.
4. Prepare later work to read and write such columns from datasources and JDBC, and to apply optimizations that depend on precise timestamp types.
### Does this PR introduce *any* user-facing change?
Yes, it extends the type system of Spark SQL.
### How was this patch tested?
By extending DataTypeSuite (round-trip and precision bounds for the new types, including invalid precisions).
```
$ build/sbt "test:testOnly *DataTypeSuite"
```
Plus SparkThrowableSuite / error-json validation if error-conditions.json is updated.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #55952:
URL: https://github.com/apache/spark/pull/55952#issuecomment-4505606476
Merging to master. Thank you, @stevomitric @peter-toth for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #55952:
URL: https://github.com/apache/spark/pull/55952#discussion_r3273487562
##########
sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import org.apache.spark.annotation.Unstable
+import org.apache.spark.sql.errors.DataTypeErrors
+
+/**
+ * Timestamp with local time zone with fractional-second precision in the nanosecond-capable range
+ * (7 to 9 decimal digits). Represents a time instant analogous to `TimestampType`, but with
+ * sub-microsecond precision: valid range is [0001-01-01T00:00:00.000000000Z,
+ * 9999-12-31T23:59:59.999999999Z] in the proleptic Gregorian calendar at UTC+00:00. No time zone
+ * is stored; the session time zone is used when converting values to and from text.
+ *
+ * @param precision
+ * Number of digits of fractional seconds for this SQL type. The valid values are 7, 8, and 9
+ * where 9 means nanosecond precision.
+ *
+ * @since 4.2.0
+ */
+@Unstable
+case class TimestampLTZNanosType(precision: Int) extends DatetimeType {
Review Comment:
First of all, because the SPIP https://docs.google.com/document/d/1DeW15QueI4PdRyPm6C6jsTZFmIjbXX2j4h-Ja5W_fsg/edit?usp=sharing defines this class with such name. Probably you might ask why I named it in this way in the SPIP. So, there are a few reasons:
1. **Pairs with TimestampNTZNanosType.** Spark already has two SQL timestamp families: with local time zone (TimestampType / TIMESTAMP_LTZ) and without (TimestampNTZType / TIMESTAMP_NTZ). The nanosecond-capable types are the same split. Alone TimestampNanosType reads as “the” nano timestamp type and does not signal which semantics apply.
2. **Matches SQL and typeName.** The class backs timestamp_ltz(p). TimestampLTZNanosType lines up with TimestampNTZNanosType and with the SPIP/SQL names; TimestampNanosType would mirror neither timestamp_ntz nor the explicit TIMESTAMP_LTZ(n) surface.
3. **Consistency with how Spark names the NTZ side.** TimestampType omits “LTZ” for history (timestamp defaulted to session-local semantics), but TimestampNTZType is explicit because the second variant exists. For new APIs where both variants are first-class, being explicit on both sides avoids the ambiguity that already bites people (TimestampType vs “timestamp with TZ” in docs).
4. **Safer for pattern matches and downstream code.** Much of the codebase branches TimestampType vs TimestampNTZType. TimestampLTZNanosType + TimestampNTZNanosType extend that model predictably; TimestampNanosType would be assumed LTZ-by-analogy-to-TimestampType, which is easy to get wrong in reviews and refactors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "stevomitric (via GitHub)" <gi...@apache.org>.
stevomitric commented on code in PR #55952:
URL: https://github.com/apache/spark/pull/55952#discussion_r3272878399
##########
sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala:
##########
@@ -275,4 +275,18 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
messageParameters = Map("precision" -> precision.toString),
cause = null)
}
+
+ def unsupportedTimestampNtzPrecisionError(precision: String): Throwable = {
Review Comment:
Why is precision a string here?
##########
sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import org.apache.spark.annotation.Unstable
+import org.apache.spark.sql.errors.DataTypeErrors
+
+/**
+ * Timestamp with local time zone with fractional-second precision in the nanosecond-capable range
+ * (7 to 9 decimal digits). Represents a time instant analogous to `TimestampType`, but with
+ * sub-microsecond precision: valid range is [0001-01-01T00:00:00.000000000Z,
+ * 9999-12-31T23:59:59.999999999Z] in the proleptic Gregorian calendar at UTC+00:00. No time zone
+ * is stored; the session time zone is used when converting values to and from text.
+ *
+ * @param precision
+ * Number of digits of fractional seconds for this SQL type. The valid values are 7, 8, and 9
+ * where 9 means nanosecond precision.
+ *
+ * @since 4.2.0
+ */
+@Unstable
+case class TimestampLTZNanosType(precision: Int) extends DatetimeType {
Review Comment:
The current timestamp type doesn't include "LTZ" in the name. Why not go with `TimestampNanosType` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk closed pull request #55952: [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType
URL: https://github.com/apache/spark/pull/55952
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #55952:
URL: https://github.com/apache/spark/pull/55952#discussion_r3274681715
##########
common/utils/src/main/resources/error/error-conditions.json:
##########
@@ -8534,6 +8534,18 @@
],
"sqlState" : "42000"
},
+ "UNSUPPORTED_TIMESTAMP_LTZ_PRECISION" : {
+ "message" : [
+ "The seconds precision <precision> of TIMESTAMP_LTZ is out of the supported range [7, 9]."
Review Comment:
Would it make sense to mention parameterless TIMESTAMP_LTZ as viable option for precision < 7?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #55952:
URL: https://github.com/apache/spark/pull/55952#discussion_r3273487562
##########
sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.types
+
+import org.apache.spark.annotation.Unstable
+import org.apache.spark.sql.errors.DataTypeErrors
+
+/**
+ * Timestamp with local time zone with fractional-second precision in the nanosecond-capable range
+ * (7 to 9 decimal digits). Represents a time instant analogous to `TimestampType`, but with
+ * sub-microsecond precision: valid range is [0001-01-01T00:00:00.000000000Z,
+ * 9999-12-31T23:59:59.999999999Z] in the proleptic Gregorian calendar at UTC+00:00. No time zone
+ * is stored; the session time zone is used when converting values to and from text.
+ *
+ * @param precision
+ * Number of digits of fractional seconds for this SQL type. The valid values are 7, 8, and 9
+ * where 9 means nanosecond precision.
+ *
+ * @since 4.2.0
+ */
+@Unstable
+case class TimestampLTZNanosType(precision: Int) extends DatetimeType {
Review Comment:
First of all, because the SPIP https://docs.google.com/document/d/1DeW15QueI4PdRyPm6C6jsTZFmIjbXX2j4h-Ja5W_fsg/edit?usp=sharing defines this class with such name. Probably you might ask why I named it in this way in the SPIP. So, there are a few reasons:
1. **Pairs with TimestampNTZNanosType.** Spark already has two SQL timestamp families: with local time zone (TimestampType / TIMESTAMP_LTZ) and without (TimestampNTZType / TIMESTAMP_NTZ). The nanosecond-capable types are the same split. A lone TimestampNanosType reads as “the” nano timestamp type and does not signal which semantics apply.
2. **Matches SQL and typeName.** The class backs timestamp_ltz(p). TimestampLTZNanosType lines up with TimestampNTZNanosType and with the SPIP/SQL names; TimestampNanosType would mirror neither timestamp_ntz nor the explicit TIMESTAMP_LTZ(n) surface.
3. **Consistency with how Spark names the NTZ side.** TimestampType omits “LTZ” for history (timestamp defaulted to session-local semantics), but TimestampNTZType is explicit because the second variant exists. For new APIs where both variants are first-class, being explicit on both sides avoids the ambiguity that already bites people (TimestampType vs “timestamp with TZ” in docs).
4. **Safer for pattern matches and downstream code.** Much of the codebase branches TimestampType vs TimestampNTZType. TimestampLTZNanosType + TimestampNTZNanosType extend that model predictably; TimestampNanosType would be assumed LTZ-by-analogy-to-TimestampType, which is easy to get wrong in reviews and refactors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on code in PR #55952:
URL: https://github.com/apache/spark/pull/55952#discussion_r3273514108
##########
sql/api/src/main/scala/org/apache/spark/sql/errors/DataTypeErrors.scala:
##########
@@ -275,4 +275,18 @@ private[sql] object DataTypeErrors extends DataTypeErrorsBase {
messageParameters = Map("precision" -> precision.toString),
cause = null)
}
+
+ def unsupportedTimestampNtzPrecisionError(precision: String): Throwable = {
Review Comment:
To pass any garbage from an user while parsing the type in json. The regex captures p as text. For values like "9" * 20, p.toInt throws NumberFormatException. I catch that and raise UNSUPPORTED_TIMESTAMP_*_PRECISION with the original digit string in the error (see DataTypeSuite — I do not want a bare NumberFormatException or a misleading message).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56876][SQL] Add TimestampNTZNanosType and TimestampLTZNanosType [spark]
Posted by "MaxGekk (via GitHub)" <gi...@apache.org>.
MaxGekk commented on PR #55952:
URL: https://github.com/apache/spark/pull/55952#issuecomment-4498693374
@dongjoon-hyun @cloud-fan @felixcheung @peter-toth @mridulm @sunchao This is an initial PR corresponded to the SPIP SPARK-56822 "Timestamps with nanosecond precision". It contains minimum changes to unblock parallel work on new types. Please, review it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org