This is an automated email from the ASF dual-hosted git repository.
MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f49ca1497fe8 [SPARK-57386][SQL] Render nanosecond timestamp types in HiveResult through the Types Framework
f49ca1497fe8 is described below
commit f49ca1497fe8d1a50e58f2f11457640369aa4568
Author: Maxim Gekk <ma...@gmail.com>
AuthorDate: Tue Jun 16 10:27:05 2026 +0200
[SPARK-57386][SQL] Render nanosecond timestamp types in HiveResult through the Types Framework
### What changes were proposed in this pull request?
Unify nanosecond timestamp (`TIMESTAMP_NTZ(p)` / `TIMESTAMP_LTZ(p)`) rendering in `HiveResult` onto the Types Framework, removing the inline duplicate renderer.
- `TimestampNanosTypeApiOps`: remove the `formatExternal(value, nested) = None` override so the Hive path shares each subclass's single-arg `formatExternal` renderer (the same one Row JSON uses). `nested` does not affect timestamp formatting.
- `HiveResult.toHiveStringDefault`: remove the inline `TimestampLTZNanosType` / `TimestampNTZNanosType` cases. The legacy path keeps no nanos handling, so a nanos value that somehow reaches it (only possible with the framework off, which the gating forbids) is unsupported rather than silently rendered.
- `TypeApiOps`: update the two-arg `formatExternal` scaladoc to reflect that Hive now shares the single-arg renderer.
### Why are the changes needed?
The nanosecond timestamp types are a Types Framework feature, implemented solely through it. External-value rendering for the framework is centralized in `TypeApiOps.formatExternal`, which already backs Row JSON (`Row.json` / `Row.prettyJson`).
The nanos ops previously overrode the two-arg `formatExternal(value, nested)` to return `None`, so `HiveResult` rendered nanos through inline pattern-matching in `toHiveStringDefault`. That duplicated the formatter logic and was documented in code as a temporary split "until nanos external rendering is unified across the zone-less (Row JSON) and zone-aware (Hive) paths".
The types are gated by `timestampNanosTypesEnabled = timestampNanosTypes.enabled && types.framework.enabled`, so a nanos column cannot exist while the framework is off; the inline cases are therefore dead code in that mode and redundant when the framework is on.
### Does this PR introduce _any_ user-facing change?
No. Nanos Hive output is identical: zone-aware LTZ, zone-independent NTZ, precision flooring, and trailing-zero trimming are all unchanged. Both the old inline path and the framework path ultimately call the same `TimestampFormatter` methods; the pre-flooring in the inline path is a no-op because values reaching Hive come from `executeCollectPublic()`, whose internal value is already stored floored to the column precision.
### How was this patch tested?
Existing `HiveResultSuite` SPARK-57257 tests cover precision 7/8/9, pre-1970 epochs, nested arrays/maps/structs, NULLs, and session-zone vs zone-independent rendering, and now exercise the framework path. Also ran `TimestampNanosTypeOpsSuite` and `RowJsonSuite`. All pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
Closes #56513 from MaxGekk/nanos-update-HiveResult.
Authored-by: Maxim Gekk <ma...@gmail.com>
Signed-off-by: Max Gekk <ma...@gmail.com>
---
.../spark/sql/types/ops/TimestampNanosTypeApiOps.scala | 18 +++++++-----------
.../org/apache/spark/sql/types/ops/TypeApiOps.scala | 8 ++++----
.../org/apache/spark/sql/execution/HiveResult.scala | 10 ----------
3 files changed, 11 insertions(+), 25 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
index d9b3a431f1ec..32aac4dcfd9c 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
@@ -42,6 +42,13 @@ import org.apache.spark.unsafe.types.TimestampNanosVal
* resolved session zone, while zone-less callers (Row JSON via formatExternal) accept the
* default, the session-local time zone config.
*
+ * External-value rendering (SPARK-57386): both consumers go through the single-arg
+ * [[formatExternal]] each subclass overrides - Row JSON (Row.json / Row.prettyJson) calls it
+ * directly, and the Hive result path (HiveResult.toHiveString) calls the two-arg overload whose
+ * default delegates to it (`nested` does not affect timestamp formatting). So both render the
+ * external value at the column precision rather than truncating to microseconds via the legacy
+ * path.
+ *
* Dataset encoders are wired here to the precision-aware leaves added by SPARK-57033
* (LocalDateTimeNanosEncoder / InstantNanosEncoder), so that turning on the Types Framework
* matches the legacy RowEncoder.encoderForDataTypeDefault behavior rather than regressing it.
@@ -58,17 +65,6 @@ abstract class TimestampNanosTypeApiOps extends TypeApiOps with DataTypeErrorsBa
// ==================== String Formatting ====================
- // Row JSON (Row.json / Row.prettyJson) holds the external Row value (java.time.Instant for LTZ,
- // java.time.LocalDateTime for NTZ); each subclass overrides the single-arg formatExternal to
- // render it through the same formatter as its zone-aware cast-to-string, so Row JSON shows the
- // nanosecond value rather than silently truncating to microseconds via the legacy path.
-
- // The Hive result path (HiveResult.toHiveString) renders nanosecond timestamps through its own
- // zone-aware default formatter, so return None here to fall through to it rather than to the
- // subclass single-arg rendering. This is a temporary split until nanos external rendering is
- // unified across the zone-less (Row JSON) and zone-aware (Hive) paths.
- override def formatExternal(value: Any, nested: Boolean): Option[String] = None
-
override def toSQLValue(v: Any): String = s"$sqlTypeName '${format(v)}'"
// ==================== Row Encoding ====================
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
index ab46e34c91e3..c76551aacd84 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
@@ -145,10 +145,10 @@ trait TypeApiOps extends Serializable {
* Renders an external value for Hive-style output (HiveResult.toHiveString). `nested` indicates
* whether the value appears inside an array/map/struct, which may format/quote it differently.
* Semantics mirror the single-arg overload: Some(s) is used directly, None falls back to
- * HiveResult's zone-aware legacy rendering. The default delegates to the single-arg overload;
- * override it separately when the two consumers need different behavior (e.g. the nanosecond
- * timestamp types render the external value on the zone-less Row JSON path but return None here
- * so the zone-aware Hive path renders them through its own formatter).
+ * HiveResult's zone-aware legacy rendering. The default delegates to the single-arg overload,
+ * so Hive shares the same renderer as Row JSON (e.g. the nanosecond timestamp types render the
+ * external Instant/LocalDateTime at the column precision on both paths). Override it separately
+ * only when the two consumers need different behavior.
*/
def formatExternal(value: Any, nested: Boolean): Option[String] = formatExternal(value)
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
index 25170da800ae..671451087435 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
@@ -131,16 +131,6 @@ object HiveResult extends SQLConfHelper {
case (t: Timestamp, TimestampType) => formatters.timestamp.format(t)
case (i: Instant, TimestampType) => formatters.timestamp.format(i)
case (l: LocalDateTime, TimestampNTZType) => formatters.timestamp.format(l)
- // Nanosecond-precision timestamps. The external values are `Instant` (LTZ) and
- // `LocalDateTime` (NTZ); convert to the physical `TimestampNanosVal` at the column precision
- // and render via the same formatter methods as the cast-to-string path (SPARK-57256), so the
- // output stays consistent. LTZ uses the session zone; NTZ is zone-independent.
- case (i: Instant, t: TimestampLTZNanosType) =>
- formatters.timestamp.formatNanos(
- DateTimeUtils.instantToTimestampNanos(i, t.precision), t.precision)
- case (l: LocalDateTime, t: TimestampNTZNanosType) =>
- formatters.timestamp.formatWithoutTimeZoneNanos(
- DateTimeUtils.localDateTimeToTimestampNanos(l, t.precision), t.precision)
case (bin: Array[Byte], BinaryType) => binaryFormatter(bin)
case (decimal: java.math.BigDecimal, DecimalType()) => decimal.toPlainString
case (n, _: NumericType) => n.toString
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org