You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ru...@apache.org on 2026/10/12 11:05:21 UTC

(spark) branch branch-4.x updated: [SPARK-59560][MLLIB] Reduce Word2Vec driver peak memory during training

This is an automated email from the ASF dual-hosted git repository.

zhengruifeng pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 9f6250fba443 [SPARK-59560][MLLIB] Reduce Word2Vec driver peak memory during training
9f6250fba443 is described below

commit 9f6250fba443ff3d20ab0bad41c2634ca864c5e2
Author: Ruifeng Zheng <ru...@apache.org>
AuthorDate: Mon Oct 12 19:04:32 2026 +0800

    [SPARK-59560][MLLIB] Reduce Word2Vec driver peak memory during training
    
    ### What changes were proposed in this pull request?
    
    This PR replaces the final `RDD.collect()` in MLlib Word2Vec training with a
    `SparkContext.runJob` result handler. The driver copies each completed partition's
    aggregated vectors into the global model arrays as the partition result arrives.
    
    ### Why are the changes needed?
    
    `collect()` retains aggregated vectors from every partition on the driver while the
    global Word2Vec model arrays are also resident. For large vocabularies and vector sizes,
    this increases peak driver memory and can cause an out-of-memory error.
    
    Processing partition results incrementally allows completed results to be reclaimed
    instead of retaining the complete aggregated RDD result.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Not run yet. This is a draft PR. The change only modifies how already-aggregated
    partition results are materialized on the driver.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #58846 from zhengruifeng/ml-word2vec-driver-memory-dev-6.
    
    Authored-by: Ruifeng Zheng <ru...@apache.org>
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
    (cherry picked from commit 4117871cba1f5dfa0339eda14b6682d8bab5a521)
    Signed-off-by: Ruifeng Zheng <ru...@apache.org>
---
 .../org/apache/spark/mllib/feature/Word2Vec.scala  | 24 ++++++++++++++--------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
index ffaf108493e7..ee380636c8e4 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
@@ -457,17 +457,23 @@ class Word2Vec extends Serializable with Logging {
       }.map { case (id, (vec, count)) =>
         BLAS.nativeBLAS.sscal(vectorSize, 1.0f / count, vec, 1)
         (id, vec)
-      }.collect()
-      var i = 0
-      while (i < synAgg.length) {
-        val index = synAgg(i)._1
-        if (index < vocabSize) {
-          Array.copy(synAgg(i)._2, 0, syn0Global, index * vectorSize, vectorSize)
-        } else {
-          Array.copy(synAgg(i)._2, 0, syn1Global, (index - vocabSize) * vectorSize, vectorSize)
+      }
+      val updateSyn = (_: Int, partitionSyn: Array[(Int, Array[Float])]) => {
+        var i = 0
+        while (i < partitionSyn.length) {
+          val (index, vec) = partitionSyn(i)
+          if (index < vocabSize) {
+            Array.copy(vec, 0, syn0Global, index * vectorSize, vectorSize)
+          } else {
+            Array.copy(vec, 0, syn1Global, (index - vocabSize) * vectorSize, vectorSize)
+          }
+          i += 1
         }
-        i += 1
       }
+      sc.runJob(
+        synAgg,
+        (iter: Iterator[(Int, Array[Float])]) => iter.toArray,
+        updateSyn)
       bcSyn0Global.destroy()
       bcSyn1Global.destroy()
     }


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org