You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/07/28 23:17:27 UTC

(spark) branch branch-4.x updated: [SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new fe8ca560d749 [SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer
fe8ca560d749 is described below

commit fe8ca560d749deae5cda26e32697a1623cbfc04e
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Tue Jul 28 16:17:08 2026 -0700

    [SPARK-58408][CORE] Register TimestampNanosVal in KryoSerializer
    
    ### What changes were proposed in this pull request?
    
    Register `org.apache.spark.unsafe.types.TimestampNanosVal` in `KryoSerializer`, next to `UTF8String` which reached the registration list the same way (SPARK-51790).
    
    ### Why are the changes needed?
    
    Since SPARK-57735, cached-batch statistics rows carry `TimestampNanosVal` min/max bounds for nanosecond-timestamp columns -- `TimestampNanosColumnStats` in the default cache serializer, and the Arrow cache serializer's vector-side statistics (SPARK-57268). The statistics `InternalRow` is a field of `DefaultCachedBatch` / `ArrowCachedBatch`, so it is serialized with the batch whenever the cache uses a serialized storage level.
    
    With `spark.serializer=KryoSerializer` and `spark.kryo.registrationRequired=true`, materializing such a cache fails at the first block write:
    
    ```
    com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException:
    Class is not registered: org.apache.spark.unsafe.types.TimestampNanosVal
    ```
    
    This is the same defect class as SPARK-51777 (`CachedBatch` classes) and SPARK-51790 (`UTF8String`): a new class reachable from the cached-batch object graph was introduced without a matching Kryo registration. Every other bound type in statistics rows (`UTF8String`, `Decimal`, primitives) is already registered; `TimestampNanosVal` was the only gap -- the remaining stats collectors for non-orderable types (Variant, CalendarInterval, Geometry) record only sizes, no value objects.
    
    Users with the default `registrationRequired=false` are unaffected (Kryo falls back to writing the class name). Nanosecond timestamp types are an unreleased 4.3.0 preview feature.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    New test in `CacheTableInKryoSuite` (the suite from SPARK-51777/51790, which runs with `registrationRequired=true`): persists a `TIMESTAMP_NTZ(9)` column with `DISK_ONLY`. It fails without the registration with the exception above and passes with it. `CacheTableInKryoSuite` (4 tests) and `core`'s `*KryoSerializer*` suites (81 tests) pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Yes, this pull request and its description were written by Claude Code.
    
    Closes #57607 from viirya/kryo-register-timestamp-nanos.
    
    Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
    (cherry picked from commit 177ce56c0df55e5a21a44ff7c59105d075990a13)
    Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
 .../main/scala/org/apache/spark/serializer/KryoSerializer.scala   | 5 ++++-
 .../test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala   | 8 ++++++++
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index ea7af01121a1..65f0ffcbe18e 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -47,7 +47,7 @@ import org.apache.spark.internal.io.FileCommitProtocol._
 import org.apache.spark.network.util.ByteUnit
 import org.apache.spark.scheduler.{CompressedMapStatus, HighlyCompressedMapStatus}
 import org.apache.spark.storage._
-import org.apache.spark.unsafe.types.UTF8String
+import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
 import org.apache.spark.util.{BoundedPriorityQueue, ByteBufferInputStream, NextIterator, SerializableConfiguration, SerializableJobConf, Utils}
 import org.apache.spark.util.collection.{BitSet, CompactBuffer}
 import org.apache.spark.util.io.ChunkedByteBuffer
@@ -236,6 +236,9 @@ class KryoSerializer(conf: SparkConf)
     kryo.register(classOf[ArrayBuffer[Any]])
     kryo.register(classOf[Array[Array[Byte]]])
     kryo.register(classOf[UTF8String])
+    // Nanosecond-timestamp min/max bounds inside cached-batch statistics rows
+    // (TimestampNanosColumnStats and the Arrow cache's vector-side stats).
+    kryo.register(classOf[TimestampNanosVal])
 
     // We can't load those class directly in order to avoid unnecessary jar dependencies.
     // We load them safely, ignore it if the class not found.
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
index 72a2da160547..feee2bee298f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/CacheTableInKryoSuite.scala
@@ -57,6 +57,14 @@ class CacheTableInKryoSuite extends SharedSparkSession {
     }
   }
 
+  test("SPARK-58408: TimestampNanosVal stats should be registered in KryoSerializer") {
+    // Cached-batch statistics rows carry TimestampNanosVal min/max bounds for
+    // nanosecond-timestamp columns (TimestampNanosColumnStats), so persisting such a cache
+    // through Kryo with registrationRequired must find the class registered.
+    val df = sql("SELECT CAST('2025-01-06 12:30:45.123456789' AS TIMESTAMP_NTZ(9)) AS ts")
+    assert(df.persist(StorageLevel.DISK_ONLY).count() === 1)
+  }
+
   test("SPARK-51813 DefaultCachedBatchKryoSerializer do not propagate nulls") {
     val ks = new KryoSerializer(this.sparkConf)
     val kryo = ks.newKryo()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org