You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2016/07/20 09:40:06 UTC
spark git commit: [SPARK-16440][MLLIB] Destroy broadcasted variables
even on driver
Repository: spark
Updated Branches:
refs/heads/master 75146be6b -> 0dc79ffd1
[SPARK-16440][MLLIB] Destroy broadcasted variables even on driver
## What changes were proposed in this pull request?
Forgotten broadcasted variables were persisted into a previous #PR 14153). This PR turns those `unpersist()` into `destroy()` so that memory is freed even on the driver.
## How was this patch tested?
Unit Tests in Word2VecSuite were run locally.
This contribution is done on behalf of Criteo, according to the
terms of the Apache license 2.0.
Author: Anthony Truchet <a....@criteo.com>
Closes #14268 from AnthonyTruchet/SPARK-16440.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0dc79ffd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0dc79ffd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0dc79ffd
Branch: refs/heads/master
Commit: 0dc79ffd1cbb45e69a35e3f5334c9a13290037a0
Parents: 75146be
Author: Anthony Truchet <a....@criteo.com>
Authored: Wed Jul 20 10:39:59 2016 +0100
Committer: Sean Owen <so...@cloudera.com>
Committed: Wed Jul 20 10:39:59 2016 +0100
----------------------------------------------------------------------
.../main/scala/org/apache/spark/mllib/feature/Word2Vec.scala | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/0dc79ffd/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
----------------------------------------------------------------------
diff --git a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
index 6b9c8ee..bc75646 100644
--- a/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
+++ b/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala
@@ -434,9 +434,9 @@ class Word2Vec extends Serializable with Logging {
bcSyn1Global.unpersist(false)
}
newSentences.unpersist()
- expTable.unpersist()
- bcVocab.unpersist()
- bcVocabHash.unpersist()
+ expTable.destroy()
+ bcVocab.destroy()
+ bcVocabHash.destroy()
val wordArray = vocab.map(_.word)
new Word2VecModel(wordArray.zipWithIndex.toMap, syn0Global)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org