You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2026/06/15 23:14:58 UTC
(spark) branch branch-4.x updated: [SPARK-57164][SQL][TEST] Add parser test coverage for nanosecond-capable timestamp types across data-type string entry points
This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 9a230895724a [SPARK-57164][SQL][TEST] Add parser test coverage for nanosecond-capable timestamp types across data-type string entry points
9a230895724a is described below
commit 9a230895724a9bf16b122505147ec6e5fc160f18
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Tue Jun 16 01:14:32 2026 +0200
[SPARK-57164][SQL][TEST] Add parser test coverage for nanosecond-capable timestamp types across data-type string entry points
### What changes were proposed in this pull request?
Test-only changes (sub-task of [SPARK-56822](https://issues.apache.org/jira/browse/SPARK-56822), SPIP: Timestamps with nanosecond precision) adding focused coverage that the nanosecond-capable timestamp spellings (`TIMESTAMP_NTZ(p)`, `TIMESTAMP_LTZ(p)`, and the `TIMESTAMP(p) WITH[OUT] [LOCAL] TIME ZONE` aliases, `p` in `[7, 9]`) parse consistently across the public string-to-DataType entry points. All new assertions are gated by `spark.sql.timestampNanosTypes.enabled`.
Coverage added:
- Catalyst `DataTypeParserSuite`: `DataType.fromDDL`, `StructType.fromDDL`, `StructType.add(name, String)`, and `DataType.parseTypeWithFallback` (DDL path + JSON fallback, the guard against the ANTLR and JSON parser families drifting apart).
- Catalyst `ExpressionParserSuite`: `CAST` / `TRY_CAST` to nanos types.
- Catalyst `DDLParserSuite`: `CREATE TABLE`, `ALTER TABLE ADD COLUMNS`, `ALTER COLUMN ... TYPE`, and a column `DEFAULT` declared with a nanos type.
- Catalyst `DataTypeSuite`: `typeName` re-parse leg of the Family B (JSON) round-trip.
- `ColumnExpressionSuite`: `Column.cast(String)` / `try_cast(String)` and `sessionState.sqlParser.parseDataType(String)`.
- `DataFrameReaderWriterSuite` / `DataStreamReaderWriterSuite`: `schema(String)`.
- `JsonFunctionsSuite` / `CsvFunctionsSuite` / `XmlFunctionsSuite`: `from_json` / `from_csv` / `from_xml` with nanos DDL schema strings, asserting the string resolves to the nanos type but the datasource rejects it at execution (JSON/CSV: `UNSUPPORTED_DATATYPE`; XML: `MALFORMED_RECORD_IN_PARSING` in FAILFAST mode).
### Why are the changes needed?
Spark parses data-type strings through two independent parser families (ANTLR `DataTypeAstBuilder` and the hand-maintained JSON `nameToType` in `DataType.scala`), reached through many distinct public surfaces. The nanos parsing was previously exercised mainly via `CatalystSqlParser.parseDataType`; the other public entry points had no explicit assertions, so a regression on any one of them (or drift between the two families) could go unnoticed.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the affected suites locally; all pass:
- `build/sbt 'catalyst/testOnly *DataTypeParserSuite *DataTypeSuite *ExpressionParserSuite *DDLParserSuite'`
- `build/sbt 'sql/testOnly *ColumnExpressionSuite *DataFrameReaderWriterSuite *DataStreamReaderWriterSuite *JsonFunctionsSuite *CsvFunctionsSuite *XmlFunctionsSuite'`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56514 from MaxGekk/nanos-parser-tests.
Authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Max Gekk <ma...@gmail.com>
(cherry picked from commit 3256fca131fa40a516ce68309c0ec2fed534d974)
Signed-off-by: Max Gekk <ma...@gmail.com>
---
.../spark/sql/catalyst/parser/DDLParserSuite.scala | 47 +++++++-
.../sql/catalyst/parser/DataTypeParserSuite.scala | 121 +++++++++++++++++++++
.../catalyst/parser/ExpressionParserSuite.scala | 32 +++++-
.../org/apache/spark/sql/types/DataTypeSuite.scala | 10 ++
.../apache/spark/sql/ColumnExpressionSuite.scala | 35 ++++++
.../org/apache/spark/sql/CsvFunctionsSuite.scala | 26 +++++
.../org/apache/spark/sql/JsonFunctionsSuite.scala | 29 ++++-
.../org/apache/spark/sql/XmlFunctionsSuite.scala | 29 +++++
.../test/DataStreamReaderWriterSuite.scala | 37 +++++++
.../sql/test/DataFrameReaderWriterSuite.scala | 36 +++++-
10 files changed, 397 insertions(+), 5 deletions(-)
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
index 7907d80e8b0c..2e8132926dcd 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DDLParserSuite.scala
@@ -19,17 +19,18 @@ package org.apache.spark.sql.catalyst.parser
import java.util.Locale
-import org.apache.spark.SparkThrowable
+import org.apache.spark.{SparkException, SparkThrowable}
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.analysis._
import org.apache.spark.sql.catalyst.expressions.{Add, And, EqualTo, GreaterThan, Hex, Literal}
import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.connector.catalog.IdentityColumnSpec
import org.apache.spark.sql.connector.catalog.TableChange.ColumnPosition.{after, first}
import org.apache.spark.sql.connector.expressions.{ApplyTransform, BucketTransform, ClusterByTransform, DaysTransform, FieldReference, HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform, YearsTransform}
import org.apache.spark.sql.connector.expressions.LogicalExpressions.bucket
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType, StringType, StructType, TimestampType}
+import org.apache.spark.sql.types.{DataType, Decimal, IntegerType, LongType, StringType, StructType, TimestampLTZNanosType, TimestampNTZNanosType, TimestampType}
import org.apache.spark.storage.StorageLevelMapper
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
@@ -90,6 +91,48 @@ class DDLParserSuite extends AnalysisTest {
parameters = Map("error" -> "':'", "hint" -> ""))
}
+ test("SPARK-57164: nanos timestamp types in CREATE TABLE / ALTER TABLE columns") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ // CREATE TABLE column.
+ val created = parsePlan(s"CREATE TABLE t (c $spelling) USING parquet")
+ assert(created.asInstanceOf[CreateTable].columns.head.dataType === expected)
+ // ALTER TABLE ... ADD COLUMNS.
+ val added = parsePlan(s"ALTER TABLE t ADD COLUMNS (c $spelling)")
+ assert(added.asInstanceOf[AddColumns].columnsToAdd.head.dataType === expected)
+ // ALTER TABLE ... ALTER COLUMN ... TYPE.
+ val altered = parsePlan(s"ALTER TABLE t ALTER COLUMN c TYPE $spelling")
+ assert(altered.asInstanceOf[AlterColumns].specs.head.newDataType === Some(expected))
+ }
+ }
+ // A column DEFAULT declared with a nanos type and a nanos typed-literal default.
+ val withDefault = parsePlan(
+ "CREATE TABLE t (c TIMESTAMP_NTZ(9) " +
+ "DEFAULT TIMESTAMP_NTZ '2020-01-01 00:00:00.123456789') USING parquet")
+ val colDef = withDefault.asInstanceOf[CreateTable].columns.head
+ assert(colDef.dataType === TimestampNTZNanosType(9))
+ assert(colDef.defaultValue.isDefined)
+ }
+ // With the preview flag off, a nanos column type is rejected.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ parsePlan("CREATE TABLE t (c TIMESTAMP_NTZ(9)) USING parquet")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("create/replace table - with IF NOT EXISTS") {
val sql = "CREATE TABLE IF NOT EXISTS my_tab(a INT, b STRING) USING parquet"
testCreateOrReplaceDdl(
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
index 4bb243bfe464..084bdbc460b7 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/DataTypeParserSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.catalyst.parser
import org.apache.spark.{SparkException, SparkFunSuite}
import org.apache.spark.sql.catalyst.plans.SQLHelper
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.TimestampTypes
import org.apache.spark.sql.types._
@@ -377,4 +378,124 @@ class DataTypeParserSuite extends SparkFunSuite with SQLHelper {
parameters = Map("error" -> "'-'", "hint" -> ""))
}
}
+
+ test("SPARK-57164: nanos timestamp types via DataType.fromDDL and StructType.fromDDL") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val expectedStruct = new StructType().add("c", expected)
+ assert(DataType.fromDDL(s"c $spelling") === expectedStruct)
+ assert(StructType.fromDDL(s"c $spelling") === expectedStruct)
+ // A bare type string (no column name) resolves directly to the nanos type.
+ assert(DataType.fromDDL(spelling) === expected)
+ }
+ // Bare TIMESTAMP(p) follows the session default timestamp type.
+ withSQLConf(SQLConf.TIMESTAMP_TYPE.key -> TimestampTypes.TIMESTAMP_NTZ.toString) {
+ assert(DataType.fromDDL(s"c TIMESTAMP($p)") ===
+ new StructType().add("c", TimestampNTZNanosType(p)))
+ }
+ withSQLConf(SQLConf.TIMESTAMP_TYPE.key -> TimestampTypes.TIMESTAMP_LTZ.toString) {
+ assert(DataType.fromDDL(s"c TIMESTAMP($p)") ===
+ new StructType().add("c", TimestampLTZNanosType(p)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-57164: fromDDL maps TIMESTAMP_*(6) to GA micro types and rejects bad precision") {
+ // Precision 6 maps to the GA micro types regardless of the preview flag.
+ Seq("true", "false").foreach { flag =>
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> flag) {
+ assert(DataType.fromDDL("c TIMESTAMP_NTZ(6)") ===
+ new StructType().add("c", TimestampNTZType))
+ assert(DataType.fromDDL("c TIMESTAMP_LTZ(6)") ===
+ new StructType().add("c", TimestampType))
+ }
+ }
+ // Out-of-range precision surfaces as INVALID_TIMESTAMP_PRECISION. The bare-type spelling is
+ // used so the primary parser raises the precision error (a SparkThrowable), which
+ // parseTypeWithFallback rethrows in preference to the fallback's parse error.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ Seq("TIMESTAMP_NTZ" -> "TIMESTAMP_NTZ", "TIMESTAMP_LTZ" -> "TIMESTAMP_LTZ").foreach {
+ case (spelling, errorType) =>
+ Seq(0, 5, 10).foreach { p =>
+ checkError(
+ exception = intercept[SparkException] {
+ DataType.fromDDL(s"$spelling($p)")
+ },
+ condition = "INVALID_TIMESTAMP_PRECISION",
+ parameters = Map("precision" -> p.toString, "type" -> errorType))
+ }
+ }
+ }
+ // With the flag off, nanos precisions are rejected with FEATURE_NOT_ENABLED.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq("TIMESTAMP_NTZ(9)", "TIMESTAMP_LTZ(7)").foreach { spelling =>
+ checkError(
+ exception = intercept[SparkException] {
+ DataType.fromDDL(spelling)
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
+
+ test("SPARK-57164: nanos timestamp types via StructType.add(name, dataTypeString)") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ assert(new StructType().add("c", spelling) ===
+ new StructType().add("c", expected))
+ }
+ }
+ }
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ new StructType().add("c", "TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
+ test("SPARK-57164: parseTypeWithFallback resolves nanos types via DDL and JSON fallback") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ // DDL path: the primary parser (fromDDL) succeeds and the fallback is never consulted.
+ assert(
+ DataType.parseTypeWithFallback(
+ s"c TIMESTAMP_NTZ($p)",
+ DataType.fromDDL,
+ fallbackParser = DataType.fromJson) ===
+ new StructType().add("c", TimestampNTZNanosType(p)))
+ // JSON fallback path: the DDL parser fails on the quoted JSON name, so the JSON parser
+ // handles it. This is the single best guard against Family A (DDL) and Family B (JSON)
+ // drifting apart.
+ val jsonName = TimestampLTZNanosType(p).typeName // "timestamp_ltz(p)"
+ assert(
+ DataType.parseTypeWithFallback(
+ s"""\"$jsonName\"""",
+ parser = DataType.fromDDL,
+ fallbackParser = DataType.fromJson) === TimestampLTZNanosType(p))
+ }
+ }
+ }
}
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
index 758e7eaf2158..636d16c78615 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/parser/ExpressionParserSuite.scala
@@ -22,7 +22,7 @@ import java.util.concurrent.TimeUnit
import scala.language.implicitConversions
-import org.apache.spark.SparkThrowable
+import org.apache.spark.{SparkException, SparkThrowable}
import org.apache.spark.sql.catalyst.FunctionIdentifier
import org.apache.spark.sql.catalyst.analysis.{UnresolvedAttribute, _}
import org.apache.spark.sql.catalyst.expressions._
@@ -315,6 +315,36 @@ class ExpressionParserSuite extends AnalysisTest {
assertEqual("cast(cast(a as int) as long)", $"a".cast(IntegerType).cast(LongType))
}
+ test("SPARK-57164: CAST / TRY_CAST to nanos timestamp types") {
+ import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ assertEqual(s"cast(a as $spelling)", $"a".cast(expected))
+ assertEqual(s"try_cast(a as $spelling)",
+ Cast($"a", expected, evalMode = EvalMode.TRY))
+ }
+ }
+ }
+ // With the preview flag off, a nanos data type in a cast target is rejected.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq("cast(a as TIMESTAMP_NTZ(9))", "try_cast(a as TIMESTAMP_LTZ(7))").foreach { sql =>
+ checkError(
+ exception = intercept[SparkException](defaultParser.parseExpression(sql)),
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
+
test("function expressions") {
assertEqual("foo()", $"foo".function())
assertEqual("foo.bar()",
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
index 0334e90af68f..ff395ea70566 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DataTypeSuite.scala
@@ -1556,6 +1556,16 @@ class DataTypeSuite extends SparkFunSuite with SQLHelper {
assert(DataType.fromJson(arrayOfNanos.json) === arrayOfNanos)
val mapOfNanos = MapType(StringType, TimestampNTZNanosType(7), valueContainsNull = true)
assert(DataType.fromJson(mapOfNanos.json) === mapOfNanos)
+
+ // Family B agrees with Family A: a nanos type's typeName re-parses (as a JSON name) to
+ // the same type. For atomic types `json` is exactly the quoted `typeName`, so this pins
+ // the type-name spelling that Family A produces against Family B's JSON parser.
+ variants.foreach { case (_, _, factory) =>
+ TimestampLTZNanosType.MIN_PRECISION to TimestampLTZNanosType.MAX_PRECISION foreach { n =>
+ val t = factory(n)
+ assert(DataType.fromJson("\"" + t.typeName + "\"") === t)
+ }
+ }
}
// Bare names without parens still map to the legacy single-precision types, regardless
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
index 919a1a258d9b..379b7320a299 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ColumnExpressionSuite.scala
@@ -31,6 +31,7 @@ import org.apache.spark.sql.UpdateFieldsBenchmark._
import org.apache.spark.sql.catalyst.expressions.{InSet, Literal, NamedExpression}
import org.apache.spark.sql.catalyst.util.DateTimeTestUtils.{outstandingTimezonesIds, outstandingZoneIds}
import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.ProjectExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -3164,4 +3165,38 @@ class ColumnExpressionSuite extends SharedSparkSession {
Seq(20, 40, 60).toDF()
)
}
+
+ test("SPARK-57164: Column.cast/try_cast(String) and sessionState parser for nanos types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ // Column.cast(String) / Column.try_cast(String) parse the type at call time.
+ assert($"id".cast(spelling).expr.dataType === expected)
+ assert($"id".try_cast(spelling).expr.dataType === expected)
+ // The programmatic catalog-string entry point.
+ assert(spark.sessionState.sqlParser.parseDataType(spelling) === expected)
+ }
+ }
+ }
+ // With the preview flag off, the string overloads reject the nanos spelling at call time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ Seq[() => Any](
+ () => $"id".cast("TIMESTAMP_NTZ(9)"),
+ () => $"id".try_cast("TIMESTAMP_LTZ(7)"),
+ () => spark.sessionState.sqlParser.parseDataType("TIMESTAMP_NTZ(9)")).foreach { f =>
+ checkError(
+ exception = intercept[SparkException](f()),
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+ }
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
index 70dbb7c8ce07..a70ef16bdf71 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CsvFunctionsSuite.scala
@@ -26,6 +26,7 @@ import scala.jdk.CollectionConverters._
import org.apache.spark.{SparkException, SparkRuntimeException,
SparkUnsupportedOperationException, SparkUpgradeException}
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.errors.DataTypeErrors.toSQLType
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -47,6 +48,31 @@ class CsvFunctionsSuite extends SharedSparkSession {
Row(Row(1)) :: Nil)
}
+ test("SPARK-57164: from_csv with a nanos timestamp DDL schema string") {
+ val df = Seq("2020-01-01T00:00:00.123456789").toDF("value")
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(
+ from_csv($"value", lit(s"c $spelling"), Map.empty[String, String].asJava).as("v"))
+ // The schema string resolves to the nanos type ...
+ assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType === expected)
+ // ... but the CSV datasource does not support nanosecond timestamps yet, so the
+ // value converter rejects it at execution.
+ checkError(
+ exception = intercept[SparkUnsupportedOperationException](parsed.collect()),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> toSQLType(expected)))
+ }
+ }
+ }
+ }
+
test("from_csv with non struct schema") {
checkError(
exception = intercept[AnalysisException] {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
index 6b50dcd0871b..8857a9977b57 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala
@@ -23,10 +23,12 @@ import java.util.Locale
import scala.jdk.CollectionConverters._
-import org.apache.spark.{SparkException, SparkRuntimeException}
+import org.apache.spark.{SparkException, SparkRuntimeException, SparkUnsupportedOperationException}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.Cast._
import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
+import org.apache.spark.sql.errors.DataTypeErrors.toSQLType
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -365,6 +367,31 @@ class JsonFunctionsSuite extends SharedSparkSession {
Row(Row(1, "haa")) :: Nil)
}
+ test("SPARK-57164: from_json with a nanos timestamp DDL schema string") {
+ val df = Seq("""{"c": "2020-01-01T00:00:00.123456789"}""").toDF("value")
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(
+ from_json($"value", s"c $spelling", Map.empty[String, String]).as("v"))
+ // The schema string resolves to the nanos type ...
+ assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType === expected)
+ // ... but the JSON datasource does not support nanosecond timestamps yet, so the
+ // value converter rejects it at execution.
+ checkError(
+ exception = intercept[SparkUnsupportedOperationException](parsed.collect()),
+ condition = "UNSUPPORTED_DATATYPE",
+ parameters = Map("typeName" -> toSQLType(expected)))
+ }
+ }
+ }
+ }
+
test("from_json with NULL in options map values") {
val df = Seq("""{"str": "World"}""").toDS()
val schema = "str STRING"
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
index d07758721fca..f87bd2df5164 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/XmlFunctionsSuite.scala
@@ -22,6 +22,8 @@ import java.util.Locale
import scala.jdk.CollectionConverters._
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.WholeStageCodegenExec
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
@@ -40,6 +42,33 @@ class XmlFunctionsSuite extends SharedSparkSession {
Row(Row(1)) :: Nil)
}
+ test("SPARK-57164: from_xml with a nanos timestamp DDL schema string") {
+ val df = Seq("""<ROW><c>2020-01-01T00:00:00.123456789</c></ROW>""").toDF("value")
+ // FAILFAST so the value-converter rejection propagates instead of becoming a corrupt record.
+ val options = Map("mode" -> "FAILFAST").asJava
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ val parsed = df.select(from_xml($"value", s"c $spelling", options).as("v"))
+ // The schema string resolves to the nanos type ...
+ assert(parsed.schema("v").dataType.asInstanceOf[StructType]("c").dataType === expected)
+ // ... but the XML datasource does not support nanosecond timestamps yet, so the
+ // value converter rejects it at execution (surfaced as a malformed record in
+ // FAILFAST mode).
+ checkError(
+ exception = intercept[SparkException](parsed.collect()),
+ condition = "MALFORMED_RECORD_IN_PARSING.WITHOUT_SUGGESTION",
+ parameters = Map("badRecord" -> "[null]", "failFastMode" -> "FAILFAST"))
+ }
+ }
+ }
+ }
+
test("SPARK-48300: from_xml - Codegen Support") {
withTempView("XmlToStructsTable") {
val dataDF = Seq("""<ROW><a>1</a></ROW>""").toDF("value")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
index 2a186a9296f4..a1dfefe3fb89 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/streaming/test/DataStreamReaderWriterSuite.scala
@@ -29,8 +29,10 @@ import org.mockito.ArgumentMatchers.{any, eq => meq}
import org.mockito.Mockito._
import org.scalatest.BeforeAndAfter
+import org.apache.spark.SparkException
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.classic.ClassicConversions.castToImpl
import org.apache.spark.sql.classic.Dataset.ofRows
import org.apache.spark.sql.execution.datasources.DataSourceUtils
@@ -163,6 +165,41 @@ class DataStreamReaderWriterSuite extends StreamTest with BeforeAndAfter {
.stop()
}
+ test("SPARK-57164: DataStreamReader.schema(String) parses nanos timestamp types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ LastOptions.clear()
+ spark.readStream
+ .format("org.apache.spark.sql.streaming.test")
+ .schema(s"c $spelling")
+ .load()
+ val captured = LastOptions.schema.get
+ assert(captured("c").dataType === expected)
+ }
+ }
+ }
+ // With the preview flag off, schema(String) rejects the nanos spelling at parse time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ spark.readStream
+ .format("org.apache.spark.sql.streaming.test")
+ .schema("c TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("options") {
val map = new java.util.HashMap[String, String]
map.put("opt3", "3")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
index 37dd0fc3871e..cf05be197e24 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/test/DataFrameReaderWriterSuite.scala
@@ -32,13 +32,14 @@ import org.apache.parquet.schema.PrimitiveType.PrimitiveTypeName
import org.apache.parquet.schema.Type.Repetition
import org.scalatest.BeforeAndAfter
-import org.apache.spark.{SparkContext, SparkIllegalArgumentException, TestUtils}
+import org.apache.spark.{SparkContext, SparkException, SparkIllegalArgumentException, TestUtils}
import org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
import org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
import org.apache.spark.scheduler.{SparkListener, SparkListenerJobStart}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.plans.logical.{AppendData, LogicalPlan, OverwriteByExpression}
+import org.apache.spark.sql.catalyst.util.TimestampNanosTestUtils.foreachNanosPrecision
import org.apache.spark.sql.execution.QueryExecution
import org.apache.spark.sql.execution.command.DataWritingCommandExec
import org.apache.spark.sql.execution.datasources.{DataSourceUtils, HadoopFsRelation, LogicalRelation}
@@ -197,6 +198,39 @@ class DataFrameReaderWriterSuite extends SharedSparkSession with BeforeAndAfter
.save()
}
+ test("SPARK-57164: DataFrameReader.schema(String) parses nanos timestamp types") {
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
+ foreachNanosPrecision { p =>
+ Seq(
+ s"TIMESTAMP_NTZ($p)" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP_LTZ($p)" -> TimestampLTZNanosType(p),
+ s"TIMESTAMP($p) WITHOUT TIME ZONE" -> TimestampNTZNanosType(p),
+ s"TIMESTAMP($p) WITH LOCAL TIME ZONE" -> TimestampLTZNanosType(p)).foreach {
+ case (spelling, expected) =>
+ LastOptions.clear()
+ spark.read
+ .format("org.apache.spark.sql.test")
+ .schema(s"c $spelling")
+ .load()
+ val captured = LastOptions.schema.get
+ assert(captured("c").dataType === expected)
+ }
+ }
+ }
+ // With the preview flag off, schema(String) rejects the nanos spelling at parse time.
+ withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkException] {
+ spark.read.format("org.apache.spark.sql.test").schema("c TIMESTAMP_NTZ(9)")
+ },
+ condition = "FEATURE_NOT_ENABLED",
+ parameters = Map(
+ "featureName" -> "Nanosecond-precision timestamp types",
+ "configKey" -> "spark.sql.timestampNanosTypes.enabled",
+ "configValue" -> "true"))
+ }
+ }
+
test("options") {
val map = new java.util.HashMap[String, String]
map.put("opt3", "3")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org