(spark) branch branch-4.x updated: [SPARK-58688][ML] Optimize DCT transform closure

[ru...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new ad7fe1a0fa43 [SPARK-58688][ML] Optimize DCT transform closure
ad7fe1a0fa43 is described below

commit ad7fe1a0fa438c3d1a8c11aa01a6024f2c8b7056
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Aug 10 20:12:39 2026 +0800

    [SPARK-58688][ML] Optimize DCT transform closure
    
    ### What changes were proposed in this pull request?
    
    Select a dedicated forward or inverse transform function from the configured inverse-DCT flag. The resulting function captures neither the DCT transformer nor the flag.
    
    The serialized inverse transform function decreases from 2,863 bytes to 1,094 bytes (61.8%).
    
    ### Why are the changes needed?
    
    Avoiding the transformer capture reduces the memory retained by the transform closure, which is especially useful for Spark Connect server workloads.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Compiled the edited DCT source and serialized its generated transform function with Java serialization. The existing DCTSuite covers the unchanged transform behavior.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Codex (GPT-5)
    
    Closes #57889 from zhengruifeng/SPARK-58688-dct-transform-state.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
    (cherry picked from commit c32a77ae17a4120f45925e7524eca987b07f9ff3)
    Signed-off-by: Ruifeng Zheng <ru...@foxmail.com>
---
 .../scala/org/apache/spark/ml/feature/DCT.scala    | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala b/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
index fe240557c0fa..d6e66d83aed6 100644
--- a/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
+++ b/mllib/src/main/scala/org/apache/spark/ml/feature/DCT.scala
@@ -62,11 +62,23 @@ class DCT @Since("1.5.0") (@Since("1.5.0") override val uid: String)
 
   setDefault(inverse -> false)
 
-  override protected def createTransformFunc: Vector => Vector = { vec =>
-    val result = vec.toArray
-    val jTransformer = new DoubleDCT_1D(result.length)
-    if ($(inverse)) jTransformer.inverse(result, true) else jTransformer.forward(result, true)
-    Vectors.dense(result)
+  override protected def createTransformFunc: Vector => Vector = {
+    $(inverse) match {
+      case true =>
+        (vec: Vector) => {
+          val result = vec.toArray
+          val jTransformer = new DoubleDCT_1D(result.length)
+          jTransformer.inverse(result, true)
+          Vectors.dense(result)
+        }
+      case false =>
+        (vec: Vector) => {
+          val result = vec.toArray
+          val jTransformer = new DoubleDCT_1D(result.length)
+          jTransformer.forward(result, true)
+          Vectors.dense(result)
+        }
+    }
   }
 
   override protected def validateInputType(inputType: DataType): Unit = {


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org