You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/19 20:35:54 UTC
(spark) branch branch-4.x updated: [SPARK-60314][SQL] Make `Literal.sql` preserve the precision of `TIME(7)`-`TIME(9)` literals
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 930aaf1b6585 [SPARK-60314][SQL] Make `Literal.sql` preserve the precision of `TIME(7)`-`TIME(9)` literals
930aaf1b6585 is described below
commit 930aaf1b6585ebcd49d12feeec75ac2f13ea4a71
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Mon Oct 19 13:35:35 2026 -0700
[SPARK-60314][SQL] Make `Literal.sql` preserve the precision of `TIME(7)`-`TIME(9)` literals
### What changes were proposed in this pull request?
This PR makes `Literal.sql` render the fractional seconds of a `TIME(p)` literal with `p` in `[7, 9]` with exactly `p` digits. Like the nanosecond-precision timestamp literals (SPARK-57339, #56494), it truncates the value to `p` digits before formatting and pads back the trailing zeros with the existing private `padToNanosPrecision` helper. The truncation matters only for a literal whose value has more fractional digits than `p`, e.g. `Literal.create(LocalTime.of(12, 0, 0, 123456789), [...]
`TIME(0)`-`TIME(6)` literals and `Literal.toString` (display-only, trailing zeros stripped) are unchanged. `TIME(0)`-`TIME(5)` literals and `TIME(6)` literals carrying sub-microsecond digits (e.g. `lit(LocalTime.of(23, 59, 59, 999999999))`) do not round-trip through `Literal.sql` either, but they are out of the scope of this PR and will be handled in a new JIRA issue.
### Why are the changes needed?
`Literal.sql` renders a `TIME` literal through `FractionTimeFormatter`, which strips trailing fractional zeros. Since SPARK-57551 (#56622), the parser derives `TIME(7)`-`TIME(9)` from the number of fractional digits of a `TIME` literal and gives `TIME(6)` to a literal with 6 or fewer digits. So a `TIME(9)` literal whose value ends with zeros, e.g. `12:00:00.100000000`, is rendered as `TIME '12:00:00.1'` and parses back as `TIME(6)`.
- https://github.com/apache/spark/pull/56622
This causes wrong query results. By default (`spark.sql.legacy.parameterSubstitution.constantsOnly=false`), `spark.sql(sqlText, args)` and `EXECUTE IMMEDIATE ... USING` substitute each argument into the SQL text as `Literal.sql` and parse the result. So a `TIME(9)` argument becomes `TIME(6)`, and a string compared or combined with it is cast to `TIME(6)`, i.e. truncated to microseconds. For example, with `val t = expr("TIME '12:00:00.100000000'")`:
| Query | Before | After |
| --- | --- | --- |
| `spark.sql("SELECT :t = '12:00:00.100000001'", Map("t" -> t))` | `true` | `false` |
| `spark.sql("SELECT ? < '12:00:00.100000001'", Array(t))` | `false` | `true` |
| `spark.sql("SELECT s FROM VALUES ('12:00:00.100000001'), ('12:00:00.1') AS t(s) WHERE s = :t", Map("t" -> t))` | `12:00:00.100000001`, `12:00:00.1` | `12:00:00.1` |
| `EXECUTE IMMEDIATE 'SELECT ? = \'12:00:00.100000001\'' USING TIME '12:00:00.100000000'` | `true` | `false` |
| `spark.sql("SELECT CASE WHEN id = 0 THEN :t ELSE '12:00:00.123456789' END FROM range(1, 2)", Map("t" -> t))` | `12:00:00.123456` (`TIME(6)`) | `12:00:00.123456789` (`TIME(9)`) |
In addition, `Literal.sql` is used for the `EXISTS_DEFAULT` column metadata, the auto-generated column names, and the values in error messages. `EXISTS_DEFAULT` keeps the value because it is cast back to the column type, but the column names and the error messages drop the digits written in the literal.
### Does this PR introduce _any_ user-facing change?
Yes. This fixes the wrong results above. In addition, the auto-generated column names and the values in error messages keep all `p` fractional digits of a `TIME(7)`-`TIME(9)` literal.
```scala
spark.sql("SELECT TIME '12:00:00.100000000'").schema.simpleString
spark.sql("SELECT CAST(TIME '23:59:59.000000000' AS TINYINT)").collect()
```
**BEFORE**
```
struct<TIME '12:00:00.1':time(9)>
[CAST_OVERFLOW] The value TIME '23:59:59' of the type "TIME(9)" cannot be cast to "TINYINT" due to an overflow. ...
```
**AFTER**
```
struct<TIME '12:00:00.100000000':time(9)>
[CAST_OVERFLOW] The value TIME '23:59:59.000000000' of the type "TIME(9)" cannot be cast to "TINYINT" due to an overflow. ...
```
The behavior changes only for `TIME(7)`-`TIME(9)` literals whose values end with fractional zeros or have more fractional digits than the precision. This is not a user-facing change compared to the released Apache Spark versions because `TIME(7)`-`TIME(9)` was added by SPARK-57551, which exists only in the unreleased `master`, `branch-4.x`, and `branch-4.3` (Apache Spark 4.3.0 release candidates, [v4.3.0-rc1](https://github.com/apache/spark/releases/tag/v4.3.0-rc1) (2026-08-31) and [v [...]
### How was this patch tested?
Pass the CIs with the newly added test cases.
- `LiteralExpressionSuite` checks the `Literal.sql` round trip of `TIME(7)`, `TIME(8)`, and `TIME(9)` literals with trailing fractional zeros, a whole second, and no trailing zeros, and the truncation of a value with more fractional digits than the precision.
- `ParametersSuite` checks the comparison and filter results above with named and positional arguments and `EXECUTE IMMEDIATE`, and the result types of `ARRAY`, `STRUCT`, and `MAP` arguments of `TIME(9)`.
Without this fix, they fail like the following.
```
[info] - SPARK-60314: TimeType(7-9) sql keeps the precision on a round trip *** FAILED ***
[info] TimeType(6) did not equal TimeType(7) (LiteralExpressionSuite.scala:729)
```
```
[info] - SPARK-60314: TIME(7)-TIME(9) literal argument keeps its precision *** FAILED ***
[info] TimeType(6) did not equal TimeType(9) (ParametersSuite.scala:533)
```
The existing golden files are unchanged because all `TIME(7)`-`TIME(9)` literals there have no trailing fractional zeros.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59571 from dongjoon-hyun/SPARK-60314.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit dbc16eb00ac022d4d832386b5bf2725c475b45d8)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/sql/catalyst/expressions/literals.scala | 6 ++++
.../expressions/LiteralExpressionSuite.scala | 26 ++++++++++++++++
.../org/apache/spark/sql/ParametersSuite.scala | 36 ++++++++++++++++++++--
3 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
index b0cdbcc321c1..2c06ec4bdc92 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/literals.scala
@@ -688,6 +688,12 @@ case class Literal (value: Any, dataType: DataType) extends LeafExpression {
case (v: Decimal, t: DecimalType) => s"${v}BD"
case (v: Int, DateType) =>
s"DATE '$toString'"
+ // The parser derives TIME(7)..TIME(9) from the number of fractional digits (6 or fewer give
+ // TIME(6)), so render exactly `precision` digits: truncate the value like formatNanos does
+ // for the nanosecond timestamps, and pad back the trailing zeros the formatter strips.
+ case (v: Long, t: TimeType) if t.precision > TimeType.MICROS_PRECISION =>
+ val truncated = DateTimeUtils.truncateTimeToPrecision(v, t.precision)
+ s"TIME '${padToNanosPrecision(new FractionTimeFormatter().format(truncated), t.precision)}'"
case (_: Long, _: TimeType) =>
s"TIME '$toString'"
case (v: Long, TimestampType) =>
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
index 85784579ad6d..323f696485bc 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/LiteralExpressionSuite.scala
@@ -717,6 +717,32 @@ class LiteralExpressionSuite extends SparkFunSuite with ExpressionEvalHelper {
}
}
+ test("SPARK-60314: TimeType(7-9) sql keeps the precision on a round trip") {
+ (TimeType.MICROS_PRECISION + 1 to TimeType.MAX_PRECISION).foreach { p =>
+ Seq(
+ "12:00:00.1" + "0" * (p - 1),
+ "12:00:00." + "0" * p,
+ "23:59:59.123456789".take(9 + p)
+ ).foreach { str =>
+ val lit = Literal(DateTimeUtils.localTimeToNanos(LocalTime.parse(str)), TimeType(p))
+ val parsed = Literal.fromSQL(lit.sql)
+ assert(parsed.dataType === TimeType(p))
+ assert(parsed === lit)
+ assert(lit.sql === s"TIME '$str'")
+ }
+ }
+
+ // A value with more fractional digits than the precision is truncated to the precision.
+ val nanos = DateTimeUtils.localTimeToNanos(LocalTime.of(23, 59, 59, 123456789))
+ Seq(7, 8).foreach { p =>
+ val str = "23:59:59.123456789".take(9 + p)
+ val lit = Literal(nanos, TimeType(p))
+ assert(lit.sql === s"TIME '$str'")
+ assert(Literal.fromSQL(lit.sql) ===
+ Literal(DateTimeUtils.localTimeToNanos(LocalTime.parse(str)), TimeType(p)))
+ }
+ }
+
test("context independent foldable literals") {
val array = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType))
val map = Literal.create(
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
index e3c98d01db25..ffe685343375 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/ParametersSuite.scala
@@ -27,10 +27,10 @@ import org.apache.spark.sql.catalyst.plans.logical.{CacheTableAsSelect, CTEInChi
import org.apache.spark.sql.catalyst.trees.SQLQueryContext
import org.apache.spark.sql.catalyst.trees.TreePattern.PARAMETER
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
-import org.apache.spark.sql.functions.{array, call_function, lit, map, map_from_arrays, map_from_entries, str_to_map, struct}
+import org.apache.spark.sql.functions.{array, call_function, expr, lit, map, map_from_arrays, map_from_entries, str_to_map, struct}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
-import org.apache.spark.sql.types.{CharType, DataType, DecimalType, StructField, VarcharType}
+import org.apache.spark.sql.types.{CharType, DataType, DecimalType, StructField, TimeType, VarcharType}
class ParametersSuite extends SharedSparkSession {
@@ -526,6 +526,38 @@ class ParametersSuite extends SharedSparkSession {
}
}
+ test("SPARK-60314: TIME(7)-TIME(9) literal argument keeps its precision") {
+ // Arguments are substituted as `Literal.sql` text. A TIME(9) argument rendered without its
+ // trailing zeros is parsed as TIME(6), which truncates the strings compared with it.
+ val time = expr("TIME '12:00:00.100000000'")
+ assert(spark.sql("SELECT :t", args = Map("t" -> time)).schema.head.dataType === TimeType(9))
+ checkAnswer(
+ spark.sql("SELECT :t = '12:00:00.100000001'", args = Map("t" -> time)),
+ Row(false))
+ checkAnswer(
+ spark.sql("SELECT ? < '12:00:00.100000001'", args = Array(time)),
+ Row(true))
+ checkAnswer(
+ spark.sql(
+ "SELECT s FROM VALUES ('12:00:00.100000001'), ('12:00:00.1') AS t(s) WHERE s = :t",
+ args = Map("t" -> time)),
+ Row("12:00:00.1") :: Nil)
+ checkAnswer(
+ sql(
+ """EXECUTE IMMEDIATE 'SELECT ? = \'12:00:00.100000001\''
+ |USING TIME '12:00:00.100000000'""".stripMargin),
+ Row(false))
+ // Nested values are rendered element by element, so they keep the precision as well.
+ Seq(
+ "array(TIME '12:00:00.100000000')" -> "array<time(9)>",
+ "named_struct('f', TIME '12:00:00.100000000')" -> "struct<f:time(9)>",
+ "map(1, TIME '12:00:00.100000000')" -> "map<int,time(9)>"
+ ).foreach { case (arg, expected) =>
+ val df = spark.sql("SELECT :c", args = Map("c" -> expr(arg)))
+ assert(df.schema.head.dataType.simpleString === expected)
+ }
+ }
+
test("unused positional arguments") {
checkAnswer(
spark.sql("SELECT ?, ?", Array(1, "abc", 3.14f)),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org