You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Ishiihara <gi...@git.apache.org> on 2014/08/01 17:51:05 UTC
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
GitHub user Ishiihara opened a pull request:
https://github.com/apache/spark/pull/1719
[MLlib] word2vec: Distributed Representation of Words
Vector representation of words. This is a pull request regarding SPARK-2510 at https://issues.apache.org/jira/browse/SPARK-2510
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Ishiihara/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1719.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1719
----
commit 8d6befe21e26cc843fc96e4c2934a15c0797ce51
Author: Liquan Pei <lp...@gopivotal.com>
Date: 2014-08-01T07:45:22Z
initial commit
commit 0aafb1b02a19fe4f1689543baf1882a49a7ff11a
Author: Liquan Pei <lp...@gopivotal.com>
Date: 2014-08-01T15:34:11Z
Add comments, minor fixes
commit e4a04d32be284f9a7ab2d3f57d745342912930a7
Author: Liquan Pei <lp...@gopivotal.com>
Date: 2014-08-01T15:46:38Z
minor fix
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51020440
QA results for PR 1719:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17842/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50901696
QA results for PR 1719:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br>class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17678/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51017655
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17840/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721791
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
--- End diff --
should create an empty vector of the same size or throw an exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721704
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
--- End diff --
`_model` is not necessary. Shall we change the constructor to
~~~
class Word2VecModel private (private val model:RDD[(String, Array[Double])]) {
~~~
`Serializable` may be unnecessary.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15727095
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ // Hierarchical softmax
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match {
+ case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
--- End diff --
Should use weighted sum to handle imbalanced partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50951455
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17728/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735818
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
--- End diff --
Is `V` the name used in the original implementation? If not, we should rename it to `bcVocab`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51015705
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17840/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15720792
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
--- End diff --
and briefly explain what it does.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50955037
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17756/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15720780
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
--- End diff --
We need more docs here, for example, link to the C implementation and the original papers for word2vec.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15727134
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ // Hierarchical softmax
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match {
+ case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
--- End diff --
`syn0_2` -> `syn1_1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15740911
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import org.scalatest.FunSuite
+
+import org.apache.spark.mllib.util.LocalSparkContext
+
+class Word2VecSuite extends FunSuite with LocalSparkContext {
+
+ // TODO: add more tests
+
+ test("Word2Vec") {
+ val sentence = "a b " * 100 + "a c " * 10
+ val localDoc = Seq(sentence, sentence)
+ val doc = sc.parallelize(localDoc)
+ .map(line => line.split(" ").toSeq)
+ val size = 10
+ val startingAlpha = 0.025
+ val window = 2
+ val minCount = 2
+ val num = 2
+
+ val model = Word2Vec.train(doc, size, startingAlpha, window, minCount)
+ val syms = model.findSynonyms("a", 2)
+ assert(syms.length == num)
+ assert(syms(0)._1 == "b")
+ assert(syms(1)._1 == "c")
+ }
+
+
+ test("Word2VecModel") {
+ val num = 2
+ val localModel = Seq(
+ ("china" , Array(0.50f, 0.50f, 0.50f, 0.50f)),
--- End diff --
` , ` -> `, ` (remove extra spaces)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by Ishiihara <gi...@git.apache.org>.
Github user Ishiihara commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15723320
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
--- End diff --
word2vec is sensitive to alpha. Larger alpha may generate meaningless result
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by Ishiihara <gi...@git.apache.org>.
Github user Ishiihara commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50949833
@mengxr result of 4 and 10 partitions make sense but result of 100 partitions doesn't make sense.
Made changes according to review except the random seed. Will continue in 1 hour or so.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50998279
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17820/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51021526
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17846/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50904679
QA results for PR 1719:<br>- This patch FAILED unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br>class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17680/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721747
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
--- End diff --
`cosineSimilarity` is more accurate.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50941289
@Ishiihara Did you try using more than one partitions? How is the accuracy?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50949973
How about making more iterations?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721094
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
--- End diff --
Btw, this is definitely an experimental feature. Please add `@Experimental` tag. Example:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala#L45
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721821
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
+ else result(0)
+ }
+
+ /**
+ * Transforms an RDD to its vector representation
+ * @param dataset a an RDD of words
+ * @return RDD of vector representation
+ */
+
+ def transform(dataset: RDD[String]): RDD[Array[Double]] = {
+ dataset.map(word => transform(word))
+ }
+
+ /**
+ * Find synonyms of a word
+ * @param word a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(word: String, num: Int): Array[(String, Double)] = {
+ val vector = transform(word)
+ if (vector.isEmpty) Array[(String, Double)]()
+ else findSynonyms(vector,num)
+ }
+
+ /**
+ * Find synonyms of the vector representation of a word
+ * @param vector vector representation of a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(vector: Array[Double], num: Int): Array[(String, Double)] = {
+ require(num > 0, "Number of similar words should > 0")
+ val topK = model.map(
+ {case(w, vec) => (distance(vector, vec), w)})
--- End diff --
~~~
val topK = model.map { case (w, vec) =>
(distance(vector, vec), w)
}
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by Ishiihara <gi...@git.apache.org>.
Github user Ishiihara commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50904281
@mengxr code format done. Working on test case of algorithm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by Ishiihara <gi...@git.apache.org>.
Github user Ishiihara commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15741135
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import org.scalatest.FunSuite
+
+import org.apache.spark.mllib.util.LocalSparkContext
+
+class Word2VecSuite extends FunSuite with LocalSparkContext {
+
+ // TODO: add more tests
+
+ test("Word2Vec") {
+ val sentence = "a b " * 100 + "a c " * 10
+ val localDoc = Seq(sentence, sentence)
+ val doc = sc.parallelize(localDoc)
+ .map(line => line.split(" ").toSeq)
+ val size = 10
+ val startingAlpha = 0.025
+ val window = 2
+ val minCount = 2
+ val num = 2
+
+ val model = Word2Vec.train(doc, size, startingAlpha, window, minCount)
+ val syms = model.findSynonyms("a", 2)
+ assert(syms.length == num)
+ assert(syms(0)._1 == "b")
+ assert(syms(1)._1 == "c")
+ }
+
+
+ test("Word2VecModel") {
+ val num = 2
+ val localModel = Seq(
+ ("china" , Array(0.50f, 0.50f, 0.50f, 0.50f)),
--- End diff --
Fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721159
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
--- End diff --
Please leave a note that the variable/method names are to match the original C implementation. Then people understand why, e.g., we map `size` to `layer1Size`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50904577
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17680/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51008212
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17830/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15739777
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,426 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+import org.apache.spark.storage.StorageLevel
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]){
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def learnVocabPerPartition(words:RDD[String]) {
+
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val bcVocab = sc.broadcast(vocab)
+ val bcVocabHash = sc.broadcast(vocabHash)
+
+ val sentences: RDD[Array[Int]] = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = bcVocabHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(parallelism).cache()
+ var syn0Global
+ = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ var syn1Global = new Array[Double](vocabSize * layer1Size)
+
+ for(iter <- 1 to numIterations) {
+ val (aggSyn0, aggSyn1, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
--- End diff --
Do you mind changing it to `treeAggregate`? (`import org.apache.spark.mllib.rdd.RDDFunctions._`) I tried 64 partitions but it is slower than 8 partitions because aggregation is linear on the number of partitions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735823
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(parallelism).cache()
+ var syn0Global
+ = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ var syn1Global = new Array[Double](vocabSize * layer1Size)
+
+ for(iter <- 1 to numIterations) {
+ val (aggSyn0, aggSyn1, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
+ seqOp = (c, v) => (c, v) match {
+ case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - parallelism * wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ // Hierarchical softmax
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match {
+ case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn1_1, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+ syn0Global = aggSyn0
+ syn1Global = aggSyn1
+ }
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(syn0Global, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum)
+ .partitionBy(new HashPartitioner(modelPartitionNum)).cache()
+ new Word2VecModel(modelRDD)
--- End diff --
please call `newSentences.unpersist()` before exit
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50956096
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17756/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721232
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
--- End diff --
move `vocabulary` to previous line and add `.`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51025157
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17846/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15720956
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
--- End diff --
For each argument, we need to have a coressponding JavaDoc. For example:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala#L45
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721454
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
--- End diff --
insert an empty line between method definitions
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51023016
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17843/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703584
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.HashMap
--- End diff --
Please remove this import and use `mutable.HashMap` explicitly in the code to be specific.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51021254
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51001654
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17823/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721765
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
--- End diff --
Shall we use `mllib.linalg.Vector` as the return type?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703684
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -0,0 +1,40 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import org.scalatest.FunSuite
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.util.LocalSparkContext
+
+class Word2VecSuite extends FunSuite with LocalSparkContext {
+ test("word2vec") {
+ val num = 2
+ val localModel = Seq(
+ ("china" , Array(0.50, 0.50, 0.50, 0.50)),
+ ("japan" , Array(0.40, 0.50, 0.50, 0.50)),
+ ("taiwan", Array(0.60, 0.50, 0.50, 0.50)),
+ ("korea" , Array(0.45, 0.60, 0.60, 0.60))
+ )
+ val model = new Word2VecModel(sc.parallelize(localModel, 2))
+ val synons = model.findSynonyms("china", num)
+ assert(synons.length == num)
+ assert(synons(0)._1 == "taiwan")
+ assert(synons(1)._1 == "japan")
+ }
+}
--- End diff --
ditto: new line at the end
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721802
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
+ else result(0)
+ }
+
+ /**
+ * Transforms an RDD to its vector representation
+ * @param dataset a an RDD of words
+ * @return RDD of vector representation
+ */
+
+ def transform(dataset: RDD[String]): RDD[Array[Double]] = {
--- End diff --
The return type should be `RDD[Vector]`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51018495
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17842/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15727075
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
--- End diff --
Shall we make it configurable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50901594
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17678/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703745
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.HashMap
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
+ else result(0)
+ }
+
+ /**
+ * Transforms an RDD to its vector representation
+ * @param dataset a an RDD of words
+ * @return RDD of vector representation
+ */
+
+ def transform(dataset: RDD[String]): RDD[Array[Double]] = {
+ dataset.map(word => transform(word))
+ }
+
+ /**
+ * Find synonyms of a word
+ * @param word a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(word: String, num: Int): Array[(String, Double)] = {
+ val vector = transform(word)
+ if (vector.isEmpty) Array[(String, Double)]()
+ else findSynonyms(vector,num)
+ }
+
+ /**
+ * Find synonyms of the vector representation of a word
+ * @param vector vector representation of a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(vector: Array[Double], num: Int): Array[(String, Double)] = {
+ require(num > 0, "Number of similar words should > 0")
+ val topK = model.map(
+ {case(w, vec) => (distance(vector, vec), w)})
+ .sortByKey(ascending = false)
+ .take(num + 1)
+ .map({case (dist, w) => (w, dist)}).drop(1)
+
+ topK
+ }
+}
+
+object Word2Vec extends Serializable with Logging {
--- End diff --
`Serializable` and `Logging` are not used.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721918
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
--- End diff --
`import scala.util.Random`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703588
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.HashMap
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
--- End diff --
ditto: if not many classes are used, please list them
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721868
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
+ else result(0)
+ }
+
+ /**
+ * Transforms an RDD to its vector representation
+ * @param dataset a an RDD of words
+ * @return RDD of vector representation
+ */
+
+ def transform(dataset: RDD[String]): RDD[Array[Double]] = {
+ dataset.map(word => transform(word))
+ }
+
+ /**
+ * Find synonyms of a word
+ * @param word a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(word: String, num: Int): Array[(String, Double)] = {
+ val vector = transform(word)
+ if (vector.isEmpty) Array[(String, Double)]()
+ else findSynonyms(vector,num)
+ }
+
+ /**
+ * Find synonyms of the vector representation of a word
+ * @param vector vector representation of a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(vector: Array[Double], num: Int): Array[(String, Double)] = {
+ require(num > 0, "Number of similar words should > 0")
+ val topK = model.map(
+ {case(w, vec) => (distance(vector, vec), w)})
+ .sortByKey(ascending = false)
+ .take(num + 1)
+ .map({case (dist, w) => (w, dist)}).drop(1)
--- End diff --
~~~
.map(_.swap)
.tail // drop itself
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735816
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
--- End diff --
`sentences` => `sentences: RDD[Array[Int]]` for code readability
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703627
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.HashMap
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
--- End diff --
Force using `F2jBLAS` may be better because only level-1 operations are used. I will send you a PR on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735830
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(parallelism).cache()
+ var syn0Global
+ = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ var syn1Global = new Array[Double](vocabSize * layer1Size)
+
+ for(iter <- 1 to numIterations) {
+ val (aggSyn0, aggSyn1, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
+ seqOp = (c, v) => (c, v) match {
+ case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - parallelism * wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ // Hierarchical softmax
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match {
+ case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
--- End diff --
Using weighted sum may be more robust.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721768
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
--- End diff --
remove empty line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51000166
QA results for PR 1719:<br>- This patch PASSES unit tests.<br>- This patch merges cleanly<br>- This patch adds the following public classes (experimental):<br>class Word2Vec(<br><br>For more information see test ouptut:<br>https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17820/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15720822
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
--- End diff --
Is word2vec sensitive to alpha? If not, we should try to expose less parameters to users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721930
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
--- End diff --
Try to fix the seed to make the computation reproducible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51023600
LGTM. Merged into both master and branch-1.1. @Ishiihara Thanks a lot for implementing word2vec! Please help improve its performance during the QA period. One task left is Java support. If you want to spend some time on it, there are some examples in `HashingTF.scala` and `JavaTfIdfSuite.java`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50901119
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721563
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
--- End diff --
This line is definitely too long. Try running `sbt/sbt scalastyle` at local. It will check style issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721254
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
--- End diff --
need more information about what each record should be, a word, a sentence, or a paragraph?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51006777
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17830/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50950095
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17728/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703583
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
--- End diff --
How many methods are used from `scala.util.*`? If this is less than 4, please list them explicitly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/1719
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50900758
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721908
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -0,0 +1,40 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import org.scalatest.FunSuite
+import org.apache.spark.SparkContext._
--- End diff --
insert an empty line to separate 3rd-party imports from spark imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703716
--- Diff: mllib/src/test/scala/org/apache/spark/mllib/feature/Word2VecSuite.scala ---
@@ -0,0 +1,40 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import org.scalatest.FunSuite
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.util.LocalSparkContext
+
+class Word2VecSuite extends FunSuite with LocalSparkContext {
+ test("word2vec") {
--- End diff --
This only tests the model. Could you add a test for the algorithm?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15703665
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,376 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util._
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable.HashMap
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
+ new Word2VecModel(modelRDD)
+ }
+}
+
+/**
+* Word2Vec model
+*/
+class Word2VecModel (val _model:RDD[(String, Array[Double])]) extends Serializable {
+
+ val model = _model
+
+ private def distance(v1: Array[Double], v2: Array[Double]): Double = {
+ require(v1.length == v2.length, "Vectors should have the same length")
+ val n = v1.length
+ val norm1 = blas.dnrm2(n, v1, 1)
+ val norm2 = blas.dnrm2(n, v2, 1)
+ if (norm1 == 0 || norm2 == 0) return 0.0
+ blas.ddot(n, v1, 1, v2,1) / norm1 / norm2
+ }
+
+ /**
+ * Transforms a word to its vector representation
+ * @param word a word
+ * @return vector representation of word
+ */
+
+ def transform(word: String): Array[Double] = {
+ val result = model.lookup(word)
+ if (result.isEmpty) Array[Double]()
+ else result(0)
+ }
+
+ /**
+ * Transforms an RDD to its vector representation
+ * @param dataset a an RDD of words
+ * @return RDD of vector representation
+ */
+
+ def transform(dataset: RDD[String]): RDD[Array[Double]] = {
+ dataset.map(word => transform(word))
+ }
+
+ /**
+ * Find synonyms of a word
+ * @param word a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(word: String, num: Int): Array[(String, Double)] = {
+ val vector = transform(word)
+ if (vector.isEmpty) Array[(String, Double)]()
+ else findSynonyms(vector,num)
+ }
+
+ /**
+ * Find synonyms of the vector representation of a word
+ * @param vector vector representation of a word
+ * @param num number of synonyms to find
+ * @return array of (word, similarity)
+ */
+ def findSynonyms(vector: Array[Double], num: Int): Array[(String, Double)] = {
+ require(num > 0, "Number of similar words should > 0")
+ val topK = model.map(
+ {case(w, vec) => (distance(vector, vec), w)})
+ .sortByKey(ascending = false)
+ .take(num + 1)
+ .map({case (dist, w) => (w, dist)}).drop(1)
+
+ topK
+ }
+}
+
+object Word2Vec extends Serializable with Logging {
+ /**
+ * Train Word2Vec model
+ * @param input RDD of words
+ * @param size vectoer dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @return Word2Vec model
+ */
+ def train(
+ input: RDD[String],
+ size: Int,
+ startingAlpha: Double,
+ window: Int,
+ minCount: Int): Word2VecModel = {
+ new Word2Vec(size,startingAlpha, window, minCount).fit(input)
+ }
+}
--- End diff --
add new line to the end
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15724703
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
--- End diff --
Maybe we can suggest a reasonable default value in the doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721184
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
--- End diff --
make sure this line doesn't exceed 100 characters
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721390
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
--- End diff --
It may be better if we use `fit[S <: Iterable[String]](dataset: RDD[S])`. So users can use their own tokenizer. Example:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/feature/HashingTF.scala#L69
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by vivounicorn <gi...@git.apache.org>.
Github user vivounicorn commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-55355088
we have an initial implementation using downpour SGD,our goal is support ten million words/users,it 's really hard.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50901102
Jenkins, add to whitelist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-51019953
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17843/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50999984
QA tests have started for PR 1719. This patch merges cleanly. <br>View progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/17823/consoleFull
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735820
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
--- End diff --
same issue here. `bcVocabHash`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] word2vec: Distributed Representation o...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/1719#issuecomment-50901946
@Ishiihara This is great! Could you add the JIRA number to the title `[SPARK-####]`? I will ping you after I finish the first pass.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [MLlib] [SPARK-2510]Word2Vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15735833
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,414 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.feature
+
+import scala.util.Random
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark.annotation.Experimental
+import org.apache.spark.Logging
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * :: Experimental ::
+ * Word2Vec creates vector representation of words in a text corpus.
+ * The algorithm first constructs a vocabulary from the corpus
+ * and then learns vector representation of words in the vocabulary.
+ * The vector representation can be used as features in
+ * natural language processing and machine learning algorithms.
+ *
+ * We used skip-gram model in our implementation and hierarchical softmax
+ * method to train the model. The variable names in the implementation
+ * mathes the original C implementation.
+ *
+ * For original C implementation, see https://code.google.com/p/word2vec/
+ * For research papers, see
+ * Efficient Estimation of Word Representations in Vector Space
+ * and
+ * Distributed Representations of Words and Phrases and their Compositionality.
+ * @param size vector dimension
+ * @param startingAlpha initial learning rate
+ * @param window context words from [-window, window]
+ * @param minCount minimum frequncy to consider a vocabulary word
+ * @param parallelisum number of partitions to run Word2Vec
+ */
+@Experimental
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int,
+ val parallelism:Int = 1,
+ val numIterations:Int = 1)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(words:RDD[String]) {
+ vocab = words.map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(
+ x._1,
+ x._2,
+ new Array[Int](MAX_CODE_LENGTH),
+ new Array[Int](MAX_CODE_LENGTH),
+ 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in vocabulary.
+ * @param dataset an RDD of words
+ * @return a Word2VecModel
+ */
+
+ def fit[S <: Iterable[String]](dataset:RDD[S]): Word2VecModel = {
+
+ val words = dataset.flatMap(x => x)
+
+ learnVocab(words)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = words.mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(parallelism).cache()
+ var syn0Global
+ = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ var syn1Global = new Array[Double](vocabSize * layer1Size)
+
+ for(iter <- 1 to numIterations) {
+ val (aggSyn0, aggSyn1, _, _) =
+ // TODO: broadcast temp instead of serializing it directly
+ // or initialize the model in each executor
+ newSentences.aggregate((syn0Global.clone(), syn1Global.clone(), 0, 0))(
+ seqOp = (c, v) => (c, v) match {
+ case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - parallelism * wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ // Hierarchical softmax
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match {
+ case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn1_1, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+ syn0Global = aggSyn0
+ syn1Global = aggSyn1
+ }
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(syn0Global, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum)
+ .partitionBy(new HashPartitioner(modelPartitionNum)).cache()
--- End diff --
`.cache()` to `.persist(MEMROY_AND_DISK)`, so it won't be kicked out by later jobs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [MLlib] [SPARK-2510]word2vec: Distributed Repr...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/1719#discussion_r15721977
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala ---
@@ -0,0 +1,375 @@
+/*
+* Licensed to the Apache Software Foundation (ASF) under one or more
+* contributor license agreements. See the NOTICE file distributed with
+* this work for additional information regarding copyright ownership.
+* The ASF licenses this file to You under the Apache License, Version 2.0
+* Add a comment to this line
+* (the "License"); you may not use this file except in compliance with
+* the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+*/
+
+package org.apache.spark.mllib.feature
+
+import scala.util.{Random => Random}
+import scala.collection.mutable.ArrayBuffer
+import scala.collection.mutable
+
+import com.github.fommil.netlib.BLAS.{getInstance => blas}
+
+import org.apache.spark._
+import org.apache.spark.rdd._
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.HashPartitioner
+
+/**
+ * Entry in vocabulary
+ */
+private case class VocabWord(
+ var word: String,
+ var cn: Int,
+ var point: Array[Int],
+ var code: Array[Int],
+ var codeLen:Int
+)
+
+/**
+ * Vector representation of word
+ */
+class Word2Vec(
+ val size: Int,
+ val startingAlpha: Double,
+ val window: Int,
+ val minCount: Int)
+ extends Serializable with Logging {
+
+ private val EXP_TABLE_SIZE = 1000
+ private val MAX_EXP = 6
+ private val MAX_CODE_LENGTH = 40
+ private val MAX_SENTENCE_LENGTH = 1000
+ private val layer1Size = size
+ private val modelPartitionNum = 100
+
+ private var trainWordsCount = 0
+ private var vocabSize = 0
+ private var vocab: Array[VocabWord] = null
+ private var vocabHash = mutable.HashMap.empty[String, Int]
+ private var alpha = startingAlpha
+
+ private def learnVocab(dataset: RDD[String]) {
+ vocab = dataset.flatMap(line => line.split(" "))
+ .map(w => (w, 1))
+ .reduceByKey(_ + _)
+ .map(x => VocabWord(x._1, x._2, new Array[Int](MAX_CODE_LENGTH), new Array[Int](MAX_CODE_LENGTH), 0))
+ .filter(_.cn >= minCount)
+ .collect()
+ .sortWith((a, b)=> a.cn > b.cn)
+
+ vocabSize = vocab.length
+ var a = 0
+ while (a < vocabSize) {
+ vocabHash += vocab(a).word -> a
+ trainWordsCount += vocab(a).cn
+ a += 1
+ }
+ logInfo("trainWordsCount = " + trainWordsCount)
+ }
+
+ private def createExpTable(): Array[Double] = {
+ val expTable = new Array[Double](EXP_TABLE_SIZE)
+ var i = 0
+ while (i < EXP_TABLE_SIZE) {
+ val tmp = math.exp((2.0 * i / EXP_TABLE_SIZE - 1.0) * MAX_EXP)
+ expTable(i) = tmp / (tmp + 1)
+ i += 1
+ }
+ expTable
+ }
+
+ private def createBinaryTree() {
+ val count = new Array[Long](vocabSize * 2 + 1)
+ val binary = new Array[Int](vocabSize * 2 + 1)
+ val parentNode = new Array[Int](vocabSize * 2 + 1)
+ val code = new Array[Int](MAX_CODE_LENGTH)
+ val point = new Array[Int](MAX_CODE_LENGTH)
+ var a = 0
+ while (a < vocabSize) {
+ count(a) = vocab(a).cn
+ a += 1
+ }
+ while (a < 2 * vocabSize) {
+ count(a) = 1e9.toInt
+ a += 1
+ }
+ var pos1 = vocabSize - 1
+ var pos2 = vocabSize
+
+ var min1i = 0
+ var min2i = 0
+
+ a = 0
+ while (a < vocabSize - 1) {
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min1i = pos1
+ pos1 -= 1
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ } else {
+ min1i = pos2
+ pos2 += 1
+ }
+ if (pos1 >= 0) {
+ if (count(pos1) < count(pos2)) {
+ min2i = pos1
+ pos1 -= 1
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ } else {
+ min2i = pos2
+ pos2 += 1
+ }
+ count(vocabSize + a) = count(min1i) + count(min2i)
+ parentNode(min1i) = vocabSize + a
+ parentNode(min2i) = vocabSize + a
+ binary(min2i) = 1
+ a += 1
+ }
+ // Now assign binary code to each vocabulary word
+ var i = 0
+ a = 0
+ while (a < vocabSize) {
+ var b = a
+ i = 0
+ while (b != vocabSize * 2 - 2) {
+ code(i) = binary(b)
+ point(i) = b
+ i += 1
+ b = parentNode(b)
+ }
+ vocab(a).codeLen = i
+ vocab(a).point(0) = vocabSize - 2
+ b = 0
+ while (b < i) {
+ vocab(a).code(i - b - 1) = code(b)
+ vocab(a).point(i - b) = point(b) - vocabSize
+ b += 1
+ }
+ a += 1
+ }
+ }
+
+ /**
+ * Computes the vector representation of each word in
+ * vocabulary
+ * @param dataset an RDD of strings
+ * @return a Word2VecModel
+ */
+
+ def fit(dataset:RDD[String]): Word2VecModel = {
+
+ learnVocab(dataset)
+
+ createBinaryTree()
+
+ val sc = dataset.context
+
+ val expTable = sc.broadcast(createExpTable())
+ val V = sc.broadcast(vocab)
+ val VHash = sc.broadcast(vocabHash)
+
+ val sentences = dataset.flatMap(line => line.split(" ")).mapPartitions {
+ iter => { new Iterator[Array[Int]] {
+ def hasNext = iter.hasNext
+ def next = {
+ var sentence = new ArrayBuffer[Int]
+ var sentenceLength = 0
+ while (iter.hasNext && sentenceLength < MAX_SENTENCE_LENGTH) {
+ val word = VHash.value.get(iter.next)
+ word match {
+ case Some(w) => {
+ sentence += w
+ sentenceLength += 1
+ }
+ case None =>
+ }
+ }
+ sentence.toArray
+ }
+ }
+ }
+ }
+
+ val newSentences = sentences.repartition(1).cache()
+ val temp = Array.fill[Double](vocabSize * layer1Size)((Random.nextDouble - 0.5) / layer1Size)
+ val (aggSyn0, _, _, _) =
+ // TODO: broadcast temp instead of serializing it directly or initialize the model in each executor
+ newSentences.aggregate((temp.clone(), new Array[Double](vocabSize * layer1Size), 0, 0))(
+ seqOp = (c, v) => (c, v) match { case ((syn0, syn1, lastWordCount, wordCount), sentence) =>
+ var lwc = lastWordCount
+ var wc = wordCount
+ if (wordCount - lastWordCount > 10000) {
+ lwc = wordCount
+ alpha = startingAlpha * (1 - wordCount.toDouble / (trainWordsCount + 1))
+ if (alpha < startingAlpha * 0.0001) alpha = startingAlpha * 0.0001
+ logInfo("wordCount = " + wordCount + ", alpha = " + alpha)
+ }
+ wc += sentence.size
+ var pos = 0
+ while (pos < sentence.size) {
+ val word = sentence(pos)
+ // TODO: fix random seed
+ val b = Random.nextInt(window)
+ // Train Skip-gram
+ var a = b
+ while (a < window * 2 + 1 - b) {
+ if (a != window) {
+ val c = pos - window + a
+ if (c >= 0 && c < sentence.size) {
+ val lastWord = sentence(c)
+ val l1 = lastWord * layer1Size
+ val neu1e = new Array[Double](layer1Size)
+ //HS
+ var d = 0
+ while (d < vocab(word).codeLen) {
+ val l2 = vocab(word).point(d) * layer1Size
+ // Propagate hidden -> output
+ var f = blas.ddot(layer1Size, syn0, l1, 1, syn1, l2, 1)
+ if (f > -MAX_EXP && f < MAX_EXP) {
+ val ind = ((f + MAX_EXP) * (EXP_TABLE_SIZE / MAX_EXP / 2.0)).toInt
+ f = expTable.value(ind)
+ val g = (1 - vocab(word).code(d) - f) * alpha
+ blas.daxpy(layer1Size, g, syn1, l2, 1, neu1e, 0, 1)
+ blas.daxpy(layer1Size, g, syn0, l1, 1, syn1, l2, 1)
+ }
+ d += 1
+ }
+ blas.daxpy(layer1Size, 1.0, neu1e, 0, 1, syn0, l1, 1)
+ }
+ }
+ a += 1
+ }
+ pos += 1
+ }
+ (syn0, syn1, lwc, wc)
+ },
+ combOp = (c1, c2) => (c1, c2) match { case ((syn0_1, syn1_1, lwc_1, wc_1), (syn0_2, syn1_2, lwc_2, wc_2)) =>
+ val n = syn0_1.length
+ blas.daxpy(n, 1.0, syn0_2, 1, syn0_1, 1)
+ blas.daxpy(n, 1.0, syn1_2, 1, syn1_1, 1)
+ (syn0_1, syn0_2, lwc_1 + lwc_2, wc_1 + wc_2)
+ })
+
+ val wordMap = new Array[(String, Array[Double])](vocabSize)
+ var i = 0
+ while (i < vocabSize) {
+ val word = vocab(i).word
+ val vector = new Array[Double](layer1Size)
+ Array.copy(aggSyn0, i * layer1Size, vector, 0, layer1Size)
+ wordMap(i) = (word, vector)
+ i += 1
+ }
+ val modelRDD = sc.parallelize(wordMap, modelPartitionNum).partitionBy(new HashPartitioner(modelPartitionNum))
--- End diff --
line too wide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---