(spark) branch master updated: [SPARK-57808][SQL][DOC] Document the microsecond-only limitation of typed encoders and UDFs for nanosecond timestamps

[ma...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new b3bc6fb46607 [SPARK-57808][SQL][DOC] Document the microsecond-only limitation of typed encoders and UDFs for nanosecond timestamps
b3bc6fb46607 is described below

commit b3bc6fb46607534bef78f08abf32495bf48f22d9
Author: AgenticSpark <ji...@gmail.com>
AuthorDate: Thu Jul 2 11:55:28 2026 +0200

    [SPARK-57808][SQL][DOC] Document the microsecond-only limitation of typed encoders and UDFs for nanosecond timestamps
    
    ### What changes were proposed in this pull request?
    
    This adds a note to `docs/sql-ref-datatypes.md` under the `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)` entry. The note documents that typed encoders and typed Scala/Java UDF boundaries use the existing microsecond timestamp types for `java.time.Instant` and `java.time.LocalDateTime`, while Kryo encoders bypass the SQL type system. This is a sub-task of the SPARK-56822 nanosecond-timestamp SPIP.
    
    ### Why are the changes needed?
    
    Users can otherwise hit silent microsecond truncation when working with nanosecond timestamp values through typed encoders or typed UDFs. Per SPARK-57808, the decision is to document this limitation rather than change the typed encoder behavior. The note also points users to the schema-driven workaround: declare `TimestampLTZNanosType` or `TimestampNTZNanosType` in the schema and use the `Row` / `Encoders.row(schema)` path to preserve nanoseconds.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Documentation-only.
    
    ### How was this patch tested?
    
    Docs-only change; no tests. Verified the referenced APIs, configuration, and type names exist in the codebase: `ScalaReflection` maps `Instant` / `LocalDateTime` to microsecond `TimestampType` / `TimestampNTZType`, `Encoders.row(schema)` exists, and `spark.sql.timestampNanosTypes.enabled`, `TimestampLTZNanosType`, and `TimestampNTZNanosType` exist.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: GitHub Copilot CLI
    
    Closes #56956 from AgenticSpark/agenticspark/SPARK-57808-doc-nano-encoders.
    
    Authored-by: AgenticSpark <ji...@gmail.com>
    Signed-off-by: Max Gekk <ma...@gmail.com>
---
 docs/sql-ref-datatypes.md | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/sql-ref-datatypes.md b/docs/sql-ref-datatypes.md
index a5da7949d059..89fd79c48936 100644
--- a/docs/sql-ref-datatypes.md
+++ b/docs/sql-ref-datatypes.md
@@ -56,6 +56,7 @@ Spark SQL and DataFrames support the following data types:
   hour, minute, and second. All operations are performed without taking any time zone into account.
     - Note: TIMESTAMP in Spark is a user-specified alias associated with one of the TIMESTAMP_LTZ and TIMESTAMP_NTZ variations.  Users can set the default timestamp type as `TIMESTAMP_LTZ`(default value) or `TIMESTAMP_NTZ` via the configuration `spark.sql.timestampType`.
   - `TimestampNTZNanosType(precision)` / `TimestampLTZNanosType(precision)`: Preview nanosecond-capable variants of `TIMESTAMP_NTZ` and `TIMESTAMP_LTZ` with fractional seconds precision `precision` in `[7, 9]`. Unparameterized `TIMESTAMP`, `TIMESTAMP_NTZ`, and `TIMESTAMP_LTZ` remain microsecond types. In schema-driven Dataset/DataFrame conversion, Spark maps `TimestampNTZNanosType` to `java.time.LocalDateTime` and `TimestampLTZNanosType` to `java.time.Instant`; values with more sub-micro [...]
+    - Note: Nanosecond precision is only available through the schema-driven APIs above. Typed encoders derived from `java.time.Instant`/`java.time.LocalDateTime` (for example `ds.as[T]` for a case class and `spark.createDataFrame(Seq(...))` without an explicit schema) bind these types to the microsecond `TIMESTAMP_LTZ`/`TIMESTAMP_NTZ` types, and typed Scala/Java UDFs such as `udf((i: java.time.Instant) => ...)` coerce their inputs to microsecond precision at the UDF boundary. Kryo encod [...]
 
 * Interval types
   - `YearMonthIntervalType(startField, endField)`: Represents a year-month interval which is made up of a contiguous subset of the following fields:


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org