(spark) branch master updated: [SPARK-48708][CORE] Remove three unnecessary type registrations from `KryoSerializer`

[ya...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new baf461b5cb44 [SPARK-48708][CORE] Remove three unnecessary type registrations from `KryoSerializer`
baf461b5cb44 is described below

commit baf461b5cb443e008059d3e8d6fa3d4971b9b698
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Thu Jun 27 18:12:06 2024 +0800

    [SPARK-48708][CORE] Remove three unnecessary type registrations from `KryoSerializer`
    
    ### What changes were proposed in this pull request?
    This pr aims to remove three unnecessary type registrations from `KryoSerializer`, they include `None`, `Nil`, and  `scala.collection.immutable.$colon$colon` .
    
    ### Why are the changes needed?
    The types that have been cleaned up are already registered in `ScalaKryoInstantiator`, so there is no need to register them again in Spark.
    
    https://github.com/twitter/chill/blame/0a34e50f742ab74598c81f153cfe8ad90bf3a859/chill-scala/src/main/scala/com/twitter/chill/ScalaKryoInstantiator.scala#L238-L241
    
    ```scala
      None.getClass, // None
      classOf[Queue[_]],
      Nil.getClass, // Nil
      classOf[::[_]], // scala.collection.immutable.$colon$colon
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    No
    
    ### How was this patch tested?
    Exist test: "Bug: SPARK-10251" in `org.apache.spark.serializer.KryoSerializerSuite`
    
    ### Was this patch authored or co-authored using generative AI tooling?
    No
    
    Closes #47080 from LuciferYang/clean-kryo-register.
    
    Authored-by: yangjie01 <ya...@baidu.com>
    Signed-off-by: Kent Yao <ya...@apache.org>
---
 core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
index 2ca30cf4cd62..ec5d53e91b3e 100644
--- a/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
+++ b/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
@@ -227,10 +227,7 @@ class KryoSerializer(conf: SparkConf)
 
     // scalastyle:on
 
-    kryo.register(None.getClass)
-    kryo.register(Nil.getClass)
     kryo.register(Utils.classForName("scala.collection.immutable.ArraySeq$ofRef"))
-    kryo.register(Utils.classForName("scala.collection.immutable.$colon$colon"))
     kryo.register(Utils.classForName("scala.collection.immutable.Map$EmptyMap$"))
     kryo.register(Utils.classForName("scala.math.Ordering$Reverse"))
     kryo.register(Utils.classForName("scala.reflect.ClassTag$GenericClassTag"))


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org