You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2015/01/14 19:57:57 UTC
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
GitHub user jkbradley opened a pull request:
https://github.com/apache/spark/pull/4047
[SPARK-1405] [mllib] Latent Dirichlet Allocation (LDA) using EM
**This PR introduces an API + simple implementation for Latent Dirichlet Allocation (LDA).**
The [design doc for this PR](https://docs.google.com/document/d/1kSsDqTeZMEB94Bs4GTd0mvdAmduvZSSkpoSfn-seAzo) has been updated since I initially posted it. In particular, see the API and Planning for the Future sections.
## Goals
* Settle on a public API which may eventually include:
* more inference algorithms
* more options / functionality
* Have an initial easy-to-understand implementation which others may improve.
* This is NOT intended to support every topic model out there. However, if there are suggestions for making this extensible or pluggable in the future, that could be nice, as long as it does not complicate the API or implementation too much.
* This may not be very scalable currently. It will be important to check and improve accuracy. For correctness of the implementation, please check against the Asuncion et al. (2009) paper in the design doc.
## Sketch of contents of this PR
**Dependency: This makes MLlib depend on GraphX.**
Files and classes:
* LDA.scala (441 lines):
* class LDA (main estimator class)
* LDA.Document (text + document ID)
* LDAModel.scala (266 lines)
* abstract class LDAModel
* class LocalLDAModel
* class DistributedLDAModel
* LDAExample.scala (245 lines): script to run LDA + a simple (private) Tokenizer
* LDASuite.scala (144 lines)
Data/model representation and algorithm:
* Data/model: Uses GraphX, with term vertices + document vertices
* Algorithm: EM, following [Asuncion, Welling, Smyth, and Teh. "On Smoothing and Inference for Topic Models." UAI, 2009.](http://arxiv-web3.library.cornell.edu/abs/1205.2662v1)
* For more details, please see the description in the “DEVELOPERS NOTE” in LDA.scala
## Design notes
Please refer to the JIRA for more discussion + the [design doc for this PR](https://docs.google.com/document/d/1kSsDqTeZMEB94Bs4GTd0mvdAmduvZSSkpoSfn-seAzo)
Here, I list the main changes AFTER the design doc was posted.
Design decisions:
* logLikelihood() computes the log likelihood of the data and the current point estimate of parameters. This is different from the likelihood of the data given the hyperparameters, which would be harder to compute. I’d describe the current approach as more frequentist, whereas the harder approach would be more Bayesian.
* The current API takes Documents as token count vectors. I believe there should be an extended API taking RDD[String] or RDD[Array[String]] in a future PR. I have sketched this out in the design doc (as well as handier versions of getTopics returning Strings).
* Hyperparameters should be set differently for different inference/learning algorithms. See Asuncion et al. (2009) in the design doc for a good demonstration. I encourage good behavior via defaults and warning messages.
Items planned for future PRs:
* perplexity
* API taking Strings
## Questions for reviewers
* Should LDA be called LatentDirichletAllocation (and LDAModel be LatentDirichletAllocationModel)?
* Pro: We may someday want LinearDiscriminantAnalysis.
* Con: Very long names
* Should LDA reside in clustering? Or do we want a sub-package?
* mllib.topicmodel
* mllib.clustering.topicmodel
* Does the API seem reasonable and extensible?
* Unit tests:
* Should there be a test which checks a clustering results? E.g., train on a small, fake dataset with 2 very distinct topics/clusters, and ensure LDA finds those 2 topics/clusters. Does that sound useful or too flaky?
## Other notes
This has not been tested much for scaling. I have run it on a laptop for 200 iterations on a 5MB dataset with 1000 terms and 5 topics. Running it for 500 iterations made it fail because of GC problems. Future PRs will need to improve the scaling.
## Thanks to…
* @dlwh for the initial implementation
* + @jegonzal for some code in the initial implementation
* The many contributors towards topic model implementations in Spark which were referenced as a basis for this PR: @akopich @witgo @yinxusen @dlwh @EntilZha @jegonzal @IlyaKozlov
CC: @mengxr
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jkbradley/spark davidhall-lda
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4047.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4047
----
commit 186eba2736679cdb4072d37fcad296647c2ec1e2
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2014-12-16T23:58:36Z
Added 3 files from dlwh LDA implementation
commit 087d81d73b9c98e2e087005c896d184fe95b7431
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2015-01-12T20:34:32Z
Prepped LDA main class for PR, but some cleanups remain
commit 724e2cff12671ed21ac7d719570732b5a7eca96a
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2015-01-13T19:32:12Z
cleanups before PR
commit 10bf4d6b2f10b2bd7bda1ec9eb270ee60ad9a6b8
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2015-01-14T00:45:06Z
separated LDA models into own file. more cleanups before PR
commit c6e430867ca32ca6f409f953a2d47dd04a1e6e53
Author: Joseph K. Bradley <jo...@databricks.com>
Date: 2015-01-14T18:17:20Z
Unit tests and fixes for LDA, now ready for PR
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72265194
[Test build #26419 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26419/consoleFull) for PR 4047 at commit [`6fd1f71`](https://github.com/apache/spark/commit/6fd1f718ccef2464601256a84e99523c1f7d033f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Tokenizer(sc: SparkContext, stopwordFile: String) extends Serializable `
* ` class EMOptimizer(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501230
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
+
+ private[clustering] type TokenCount = Double
+
+ /** Term vertex IDs are {-1, -2, ..., -vocabSize} */
+ private[clustering] def term2index(term: Int): Long = -(1 + term.toLong)
+
+ private[clustering] def index2term(termIndex: Long): Int = -(1 + termIndex).toInt
+
+ private[clustering] def isDocumentVertex(v: (VertexId, _)): Boolean = v._1 >= 0
+
+ private[clustering] def isTermVertex(v: (VertexId, _)): Boolean = v._1 < 0
+
+ /**
+ * State for EM algorithm: data + parameter graph, plus algorithm parameters.
+ *
+ * @param graph EM graph, storing current parameter estimates in vertex descriptors and
+ * data (token counts) in edge descriptors.
+ * @param k Number of topics
+ * @param vocabSize Number of unique terms
+ * @param topicSmoothing "alpha"
+ * @param termSmoothing "eta"
+ */
+ private[clustering] case class LearningState(
+ graph: Graph[TopicCounts, TokenCount],
+ k: Int,
+ vocabSize: Int,
+ topicSmoothing: Double,
+ termSmoothing: Double) {
+ // TODO: Checkpoint periodically?
+ def next(): LearningState = copy(graph = step(graph))
+
+ private def step(graph: Graph[TopicCounts, TokenCount]): Graph[TopicCounts, TokenCount] = {
+ val eta = termSmoothing
+ val W = vocabSize
+ val alpha = topicSmoothing
+
+ val N_k = collectTopicTotals()
+ val sendMsg: EdgeContext[TopicCounts, TokenCount, (Boolean, TopicCounts)] => Unit =
+ (edgeContext) => {
+ // Compute N_{wj} gamma_{wjk}
+ val N_wj = edgeContext.attr
+ // E-STEP: Compute gamma_{wjk} (smoothed topic distributions), scaled by token count
+ // N_{wj}.
+ val scaledTopicDistribution: TopicCounts =
+ computePTopic(edgeContext.srcAttr, edgeContext.dstAttr, N_k, W, eta, alpha) *= N_wj
+ edgeContext.sendToDst((false, scaledTopicDistribution))
+ edgeContext.sendToSrc((false, scaledTopicDistribution))
+ }
+ // This is a hack to detect whether we could modify the values in-place.
+ // TODO: Add zero/seqOp/combOp option to aggregateMessages.
+ val mergeMsg: ((Boolean, TopicCounts), (Boolean, TopicCounts)) => (Boolean, TopicCounts) =
+ (m0, m1) => {
+ val sum =
+ if (m0._1) {
+ m0._2 += m1._2
+ } else if (m1._1) {
+ m1._2 += m0._2
+ } else {
+ m0._2 + m1._2
+ }
+ (true, sum)
+ }
+ // M-STEP: Aggregation computes new N_{kj}, N_{wk} counts.
+ val docTopicDistributions: VertexRDD[TopicCounts] =
+ graph.aggregateMessages[(Boolean, TopicCounts)](sendMsg, mergeMsg)
+ .mapValues(_._2)
+ // Update the vertex descriptors with the new counts.
+ graph.outerJoinVertices(docTopicDistributions) { (vid, oldDist, newDist) => newDist.get }
+ }
+
+ def collectTopicTotals(): TopicCounts = {
--- End diff --
Would the name "globalTopicTotals" make more sense?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69971357
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25558/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23553101
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
Thanks! That makes sense. I'll make a note in the code about adding simplex constraints to permit any alpha > 0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23730063
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
--- End diff --
OK, I'll remove Document or make it internal
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70356709
Here are some initial test results. There are 2 sets since I had run some before the updates from @mengxr and other tests after the updates.
Summary: Iterations keep getting longer. Will need to work on scalability, but it at least runs on medium-sized datasets. Updates from @mengxr improve scalability. Still need to test large numbers of topics.
## How tests were run
I ran using this branch:
[https://github.com/jkbradley/spark/tree/lda-testing].
It includes a little more instrumentation and a Timing script: [https://github.com/jkbradley/spark/blob/lda-testing/examples/src/main/scala/org/apache/spark/examples/mllib/LDATiming.scala].
I used the collection of stopwords from @dlwh here: [https://github.com/dlwh/spark/blob/feature/lda/examples/src/main/scala/org/apache/spark/examples/mllib/SimpleLatentDirichletAllocation.scala]
I ran using a (partial) dump of Wikipedia) consisting of about 4.7GB of gzipped text files.
My goal is to do a few sets of tests, scaling:
* corpus sizes: 10K, 100K, 1M
* k: 10, 100, 1K, 10K, 100K
* vocabSize: (not run yet)
I ran:
```
FOR SCALING CORPUS SIZE:
bin/spark-submit --class org.apache.spark.examples.mllib.LDATiming --master spark://MY_EC2_URL:7077 --driver-memory 20g /root/spark-git/examples/target/scala-2.10/spark-examples-1.3.0-SNAPSHOT-hadoop1.0.4.jar --corpusSizes "10000 100000 1000000 -1" --ks "100" --maxIterations 10 --topicSmoothing -1 --termSmoothing -1 --vocabSizes "1000000" --stopwordFile "stopwords.txt" "DATADIR"
SCALING K:
bin/spark-submit --class org.apache.spark.examples.mllib.LDATiming --master spark://MY_EC2_URL:7077 --driver-memory 20g /root/spark-git/examples/target/scala-2.10/spark-examples-1.3.0-SNAPSHOT-hadoop1.0.4.jar --corpusSizes "-1" --ks "10 100 1000 10000 100000" --maxIterations 10 --topicSmoothing -1 --termSmoothing -1 --vocabSizes "1000000" --stopwordFile "stopwords.txt" "DATADIR"
```
These used a 16-node EC2 cluster of r3.2xlarge machines.
## Results (before recent GC-related updates)
Take-aways: Iterations keep getting longer. Will need to work on scalability, but it at least runs on medium-sized datasets.
### Scaling corpus size
```
DATASET
Training set size: 9999 documents
Vocabulary size: 183313 terms
Training set size: 2537186 tokens
Preprocessing time: 53.03116398 sec
Finished training LDA model. Summary:
Training time: 51.766336365 sec
Training data average log likelihood: -2374.150807757336
Training times per iteration (sec):
16.301772584
2.714758941
2.681336067
2.812407396
3.067381155
3.148446287
4.091387595
4.845391099
4.800537527
5.784521297
```
Note that iteration times keep getting longer.
```
DATASET
Training set size: 99657 documents
Vocabulary size: 864755 terms
Training set size: 25372240 tokens
Preprocessing time: 56.172117053 sec
Finished training LDA model. Summary:
Training time: 272.724740335 sec
Training data average log likelihood: -2453.2238815201995
Training times per iteration (sec):
36.27088487
9.239099504
12.899834887
17.887326081
22.548736594
29.705019399
34.532178918
37.132915562
43.264158967
24.167606732
```
```
DATASET
Training set size: 998875 documents
Vocabulary size: 3718603 terms
Training set size: 255269137 tokens
Preprocessing time: 969.582218325 sec
(died)
```
### Scaling k
```
DATASET
Training set size: 4072243 documents
Vocabulary size: 1000000 terms
Training set size: 955849462 tokens
Preprocessing time: 1023.173703836 sec
Finished training LDA model. Summary:
Training time: 734.18870584 sec
Training data average log likelihood: -2487.378006538547
Training times per iteration (sec):
220.962623351
43.31892217
44.65509746
49.119503552
52.24947807
53.822309875
57.582740118
64.41201
70.151547256
72.043746927
```
(larger tests died)
## Results (after recent GC-related updates)
Main take-away: Updates from @mengxr improve scaling. (Notice the not dying on the later tests.)
### Scaling corpus size
```
DATASET
Training set size: 9999 documents
Vocabulary size: 549136 terms
Training set size: 2651999 tokens
Preprocessing time: 59.886667054 sec
Finished training LDA model. Summary:
Training time: 87.066636441 sec
Training data average log likelihood: -2987.1021154536284
Training times per iteration (sec):
25.916230397
3.348482537
3.133761688
4.325156952
5.231940702
6.117500071
7.246081989
8.584900244
9.413571911
9.114112645
```
```
DATASET
Training set size: 99657 documents
Vocabulary size: 1000000 terms
Training set size: 24106059 tokens
Preprocessing time: 79.624635494 sec
Finished training LDA model. Summary:
Training time: 295.883936257 sec
Training data average log likelihood: -2608.5841219446515
Training times per iteration (sec):
41.455679987
11.062455643
15.526668004
21.027575262
26.262190857
25.565775147
30.829831734
35.716967684
37.592023917
44.04621023
```
```
DATASET
Training set size: 998875 documents
Vocabulary size: 1000000 terms
Training set size: 235682866 tokens
Preprocessing time: 322.008531951 sec
Finished training LDA model. Summary:
Training time: 1073.726914484 sec
Training data average log likelihood: -2496.418600245705
Training times per iteration (sec):
119.644333858
41.555120562
52.719948261
64.48673763
88.892069695
100.981587858
123.62990158
150.65753992
157.688974275
168.515556567
```
```
DATASET
Training set size: 4072243 documents
Vocabulary size: 1000000 terms
Training set size: 955849462 tokens
Preprocessing time: 1110.123689033 sec
Finished training LDA model. Summary:
Training time: 4781.682695595 sec
Training data average log likelihood: -2483.61085533687
Training times per iteration (sec):
363.747503418
234.396490798
264.977904783
377.257946593
447.054876375
364.207562754
408.152587705
420.5513901
1080.746177241
813.866786165
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71934257
@EntilZha I'll see about merging my testing branch back into this now, though it's not quite ready yet b/c of issues with checkpointing: [https://github.com/apache/spark/pull/4253]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23724960
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
--- End diff --
How to infer topics of a new document? Do we want to include it in this PR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980345
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
--- End diff --
Maybe this is not necessary because it is already marked as @DeveloperApi. Btw, should it be @Experimental instead? We are expecting users to call it directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72584659
[Test build #26577 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26577/consoleFull) for PR 4047 at commit [`e3980d2`](https://github.com/apache/spark/commit/e3980d2967ae45d1477384a055abd1c9d7975dbd).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501095
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
--- End diff --
This would take an extra (optional) parameter which sets the default Learning Algorithm
```scala
private var algorithm: LDA.LearningAlgorithm.LearningAlgorithm
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980397
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
+
+ def setCheckpointDir(checkpointDir: String): this.type = {
+ this.checkpointDir = Some(checkpointDir)
+ this
+ }
+
+ def clearCheckpointDir(): this.type = {
+ this.checkpointDir = None
+ this
+ }
+
+ /**
+ * Period (in iterations) between checkpoints.
+ * @see [[getCheckpointDir]]
+ */
+ def getCheckpointInterval: Int = checkpointInterval
+
+ def setCheckpointInterval(checkpointInterval: Int): this.type = {
+ this.checkpointInterval = checkpointInterval
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, which are term (word) count vectors paired with IDs.
+ * The term count vectors are "bags of words" with a fixed-size vocabulary
+ * (where the vocabulary size is the length of the vector).
+ * Document IDs must be unique and >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[(Long, Vector)]): DistributedLDAModel = {
+ val state = LDA.initialState(documents, k, getDocConcentration, getTopicConcentration, seed,
+ checkpointDir, checkpointInterval)
+ var iter = 0
+ val iterationTimes = Array.fill[Double](maxIterations)(0)
+ while (iter < maxIterations) {
+ val start = System.nanoTime()
+ state.next()
+ val elapsedSeconds = (System.nanoTime() - start) / 1e9
+ iterationTimes(iter) = elapsedSeconds
+ iter += 1
+ }
+ state.graphCheckpointer.deleteAllCheckpoints()
+ new DistributedLDAModel(state, iterationTimes)
+ }
+
+ /** Java-friendly version of [[run()]] */
+ def run(documents: JavaRDD[(java.lang.Long, Vector)]): DistributedLDAModel = {
--- End diff --
`JavaPairRDD[java.lang.Long, Vector]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72572519
I just rebased after the checkpoint fix in [https://github.com/apache/spark/pull/4253]. I also added a test suite for PeriodicGraphCheckpointer. I'll rebase on [https://github.com/apache/spark/pull/4317] now...and that should fix the final PeriodicGraphCheckpointer test failure!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72289240
[Test build #26438 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26438/consoleFull) for PR 4047 at commit [`9f465b9`](https://github.com/apache/spark/commit/9f465b97b92f79c34776e0ae8f3a95b02fedaa3f).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72579585
[Test build #26577 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26577/consoleFull) for PR 4047 at commit [`e3980d2`](https://github.com/apache/spark/commit/e3980d2967ae45d1477384a055abd1c9d7975dbd).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71337678
@EntilZha Thanks for sharing your code! I like that it doesn't really change the model API, but I'm not quite clear on what will be public/private in the learning algorithm API since most methods/classes are public right now. Also, would you mind making a JIRA for adding Gibbs sampling for LDA and linking your code from there? Then we can discuss the implementation on that JIRA instead of here. (But if there are API changes needed in this PR, please do point those out.) Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23553575
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
The main use case I'm imagining is weighting of instances or terms. Then you'd want to store weighted term counts on edges instead of integral term counts.
I'm not too worried about loss of precision. Have you ever seen it be an issue with LDA?
I agree we'd be able to use different types for different algorithms' LearningStates.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70571532
By the way, the 2 sets of timing results above are not that comparable since I realized that I limited the vocab size in the later tests. I'm re-running tests to see if the updates from @mengxr helped.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23724969
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * WARNING: If vocabSize and k are large, this can return a large object!
+ *
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(): Array[Array[(Double, String)]] = describeTopics(vocabSize)
--- End diff --
Same here. Would `Array[Vector]` work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72573700
That last commit should be it, unless people have further comments!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72593275
@mengxr Oops, that commit hid some of my responses to your feedback above. The 2 issues remaining are (1) Who is responsible for materializing graphs? and (2) Where should the param docs appear (getters and/or setters)? (My last commit moved the docs to the setters.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72602768
Ok...now I believe that's it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23724946
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
--- End diff --
`should be indexed {...}` -> `should be unique`? Does GraphX require {0, 1, ..., } vertex indices?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72292465
Here are partial results (from a test which is still running) on a Wikipedia dump.
Training set size: 4072243 documents
Vocabulary size: 9869422 terms
Training set size: 1041734290 tokens
Preprocessing time: 247.831375878 sec
100 topics
16 EC2 r3.2xlarge workers
You can see the times for the first few iterations in the image.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70340275
[Test build #25689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25689/consoleFull) for PR 4047 at commit [`648f66c`](https://github.com/apache/spark/commit/648f66cc97eefd5698480402a5cf859c618c1467).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72580767
[Test build #26569 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26569/consoleFull) for PR 4047 at commit [`e3980d2`](https://github.com/apache/spark/commit/e3980d2967ae45d1477384a055abd1c9d7975dbd).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class EMOptimizer(`
* `class LogisticGradient(numClasses: Int) extends Gradient `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72580776
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26569/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980401
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
+
+ def setCheckpointDir(checkpointDir: String): this.type = {
+ this.checkpointDir = Some(checkpointDir)
+ this
+ }
+
+ def clearCheckpointDir(): this.type = {
+ this.checkpointDir = None
+ this
+ }
+
+ /**
+ * Period (in iterations) between checkpoints.
+ * @see [[getCheckpointDir]]
+ */
+ def getCheckpointInterval: Int = checkpointInterval
+
+ def setCheckpointInterval(checkpointInterval: Int): this.type = {
+ this.checkpointInterval = checkpointInterval
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, which are term (word) count vectors paired with IDs.
+ * The term count vectors are "bags of words" with a fixed-size vocabulary
+ * (where the vocabulary size is the length of the vector).
+ * Document IDs must be unique and >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[(Long, Vector)]): DistributedLDAModel = {
+ val state = LDA.initialState(documents, k, getDocConcentration, getTopicConcentration, seed,
+ checkpointDir, checkpointInterval)
+ var iter = 0
+ val iterationTimes = Array.fill[Double](maxIterations)(0)
+ while (iter < maxIterations) {
+ val start = System.nanoTime()
+ state.next()
+ val elapsedSeconds = (System.nanoTime() - start) / 1e9
+ iterationTimes(iter) = elapsedSeconds
+ iter += 1
+ }
+ state.graphCheckpointer.deleteAllCheckpoints()
+ new DistributedLDAModel(state, iterationTimes)
+ }
+
+ /** Java-friendly version of [[run()]] */
+ def run(documents: JavaRDD[(java.lang.Long, Vector)]): DistributedLDAModel = {
+ run(documents.rdd.map(id_counts => (id_counts._1.asInstanceOf[Long], id_counts._2)))
--- End diff --
`documents.rdd.asInstanceOf[RDD[(Long, Vector)]]` should work
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72578491
[Test build #26566 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26566/consoleFull) for PR 4047 at commit [`4ae2a7d`](https://github.com/apache/spark/commit/4ae2a7df685badae3b7a3b8e94a871186c148a78).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class EMOptimizer(`
* `case class HiveScriptIOSchema (`
* ` val trimed_class = serdeClassName.split("'")(1)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980394
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,357 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize, sum => brzSum}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx.{VertexId, EdgeContext, Graph}
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term index, term weight in topic).
--- End diff --
need to update doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by dlwh <gi...@git.apache.org>.
Github user dlwh commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r22962641
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable.ArrayBuffer
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
+
+import org.apache.spark.{SparkContext, SparkConf}
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.clustering.LDA
+import org.apache.spark.mllib.clustering.LDA.Document
+import org.apache.spark.mllib.linalg.SparseVector
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * An example Latent Dirichlet Allocation (LDA) app. Run with
+ * {{{
+ * ./bin/run-example mllib.DenseKMeans [options] <input>
--- End diff --
(rename)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23726961
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
--- End diff --
So currently, I'm indexing term vertices -1, -2, ... -numTerms. It's nice to get documents indexed 0,1,.... We could create new vertex IDs for documents and maintain a map between docIDs and vertexIDs. Do you think that would be preferable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23732311
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
--- End diff --
I see your point. We can have something similar to sklearn's `CountVectorizer`
http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html
It transforms `RDD[Array[String]]` to `RDD[Vector]` and stores the word <-> index bi-map in the model it produces. The model can provide transformations from words to indices and indices to words. Then LDA can use integer ids only.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r22963462
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,244 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable.ArrayBuffer
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
+
+import org.apache.spark.{SparkContext, SparkConf}
+import org.apache.spark.SparkContext._
+import org.apache.spark.mllib.clustering.LDA
+import org.apache.spark.mllib.clustering.LDA.Document
+import org.apache.spark.mllib.linalg.SparseVector
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * An example Latent Dirichlet Allocation (LDA) app. Run with
+ * {{{
+ * ./bin/run-example mllib.DenseKMeans [options] <input>
--- End diff --
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72594403
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26603/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23981332
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
--- End diff --
? Wait what's out of order? scala/java > non-spark > spark
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72265084
[Test build #26419 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26419/consoleFull) for PR 4047 at commit [`6fd1f71`](https://github.com/apache/spark/commit/6fd1f718ccef2464601256a84e99523c1f7d033f).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980420
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
--- End diff --
Doc?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23727215
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
--- End diff --
The original idea was to have a small PR which did not include everything. I'm OK with pushing more and more stuff into this PR...if other reviewers are.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501107
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * WARNING: If vocabSize and k are large, this can return a large object!
+ *
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(): Array[Array[(Double, String)]] = describeTopics(vocabSize)
+
+ /* TODO
+ * Compute the log likelihood of the observed tokens, given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior.
+ * - Even with the prior, this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated log likelihood of the data under this model
+ */
+ // def logLikelihood(documents: RDD[Document]): Double
+
+ /* TODO
+ * Compute the estimated topic distribution for each document.
+ * This is often called “theta” in the literature.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated topic distribution for each document.
+ * The returned RDD may be zipped with the given RDD, where each returned vector
+ * is a multinomial distribution over topics.
+ */
+ // def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)]
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Local LDA model.
+ * This model stores only the inferred topics.
+ * It may be used for computing topics for new documents, but it may give less accurate answers
+ * than the [[DistributedLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ *
+ * @param topics Inferred topics (vocabSize x k matrix).
+ */
+@DeveloperApi
+class LocalLDAModel private[clustering] (
+ private val topics: Matrix) extends LDAModel with Serializable {
+
+ import LDA._
+
+ override def k: Int = topics.numCols
+
+ override def vocabSize: Int = topics.numRows
+
+ override def topicsMatrix: Matrix = topics
+
+ override def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]] = {
+ val brzTopics = topics.toBreeze.toDenseMatrix
+ val topicSummary = Range(0, k).map { topicIndex =>
+ val topic = normalize(brzTopics(::, topicIndex), 1.0)
+ topic.toArray.zipWithIndex.sortBy(-_._1).take(maxTermsPerTopic)
+ }.toArray
+ topicSummary.map { topic =>
+ topic.map { case (weight, term) => (weight, term.toString) }
+ }
+ }
+
+ // TODO
+ // override def logLikelihood(documents: RDD[Document]): Double = ???
+
+ // TODO:
+ // override def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)] = ???
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Distributed LDA model.
+ * This model stores the inferred topics, the full training dataset, and the topic distributions.
+ * When computing topics for new documents, it may give more accurate answers
+ * than the [[LocalLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class DistributedLDAModel private[clustering] (
+ private val state: LDA.LearningState) extends LDAModel {
+
+ import LDA._
+
+ def toLocal: LocalLDAModel = new LocalLDAModel(topicsMatrix)
+
+ override def k: Int = state.k
+
+ override def vocabSize: Int = state.vocabSize
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ *
+ * WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.
+ */
+ override lazy val topicsMatrix: Matrix = {
+ // Collect row-major topics
+ val termTopicCounts: Array[(Int, TopicCounts)] =
+ state.graph.vertices.filter(_._1 < 0).map { case (termIndex, cnts) =>
+ (index2term(termIndex), cnts)
+ }.collect()
+ // Convert to Matrix
+ val brzTopics = BDM.zeros[Double](vocabSize, k)
+ termTopicCounts.foreach { case (term, cnts) =>
+ var j = 0
+ while (j < k) {
+ brzTopics(term, j) = cnts(j)
+ j += 1
+ }
+ }
+ Matrices.fromBreeze(brzTopics)
+ }
+
+ override def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]] = {
--- End diff --
Similarly
```scala
state.describeTopics(maxTermsPerTopic)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23553808
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
--- End diff --
I'm wary of adding more to the interface. It seems like a pretty simple transformation to go from one to the other (and most new users will probably just have raw documents).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70939923
There may be issues with extra stuff being caught in the closure. Iterations seem to shuffle more and more data. See attached image of web UI

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72248433
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26414/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501143
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
Actually, on second thought, this could be brought into Gibbs/EMLearningState. It doesn't need to be exposed I think at all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71552068
By the way, the slowdown on each subsequent iteration seems to be fixable by caching each new graph. Some caching goes on internally in GraphX, but it does not seem to cache everything. I'm calling cache() once on each iteration. I'm assuming the latest graph will push older graphs out of memory as needed. I'm currently running tests to make sure that happens.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70537930
I've had a good chance to look at PR while making changes to my own code. I really liked the Graph initialization code (especially the partition strategy), I was able to copy that and get a 2x boost almost across the board in computation phases.
# Questions in PR:
## Naming (LDA vs LatentDirichletAllocation)
I think that Linear Discriminant Analysis and Latent Dirichlet Allocation are different enough that it doesn't warrant making the class names significantly longer, so LDA is probably good.
## Package Name
I think a separate topicmodeling package is fitting, it makes it very clear that LDA is for topic modeling. The second motivation for a topicmodeling package is that it looks like there will be this EM implementation and soon the Gibbs version I am using, so both could fit in this namespace well. Along those same lines, perhaps keep LDAModel for the abstract class and for implementations use some way to denote whether it is Gibbs ( LDAGibbs, GibbsLDA, LDAWithGibbs) or EM based (LDAEM, EMLDA, LDAWithEM). Naming is hard...
## Unit Test
On tests, I am for including a small training example, similar to here:
https://github.com/EntilZha/spark/blob/LDA/graphx/src/test/scala/org/apache/spark/graphx/lib/LDASuite.scala#L52-L76
It has been very helpful to have a sanity check test as I have been working on potentially breaking changes. In that example, the data set is 9 lines, so it would probably be better to refactor and place it in an Array[String] defined in code instead of an external file.
Lastly, it looks like the abstract classes/traits haven't made it in yet to have multiple implementations satisfy a common LDA API. I have taken a careful look at the design doc and your code, and am fairly confident that I have a way to do this with minimal code changes. I will post something once I refactor what I have to satisfy it, and post the proposed changes that would need to be made. The general sketch is to make LDA a trait, LearningState a trait, and have implementations have a signature something like object LDAGibbs extends LDA and provide a LearningState implementation. More soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72140297
[Test build #26353 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26353/consoleFull) for PR 4047 at commit [`a172ef4`](https://github.com/apache/spark/commit/a172ef4fa50745a33200c206ce3106a5b03cd2c3).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980407
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
--- End diff --
Another option is to use `sc.checkpointDir` if it is available. Then the `checkpointInterval` can decide whether to checkpoint or not. Because `checkpointDir` is like a global configuration of a Spark context, it is a little strange to modify it in an algorithm, for example, what if a user changes `sc.checkpointDir` while LDA is running.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72594975
Test failure with system...will test again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501392
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
Fair enough. Reading the docs should be enough anyway (I just know I had to look up the mapping of alpha/beta to those to be 100% certain I had it right)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501384
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
Sounds good : ) I like having Double since we may want to support weighted documents at some point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72584774
Sounds great, once its merged in I will start working on Gibbs using that as a starting point. Great work!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23981203
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
+ * - Persist new graph (if not yet persisted), and put in queue of persisted graphs.
+ * - Unpersist graphs from queue until there are at most 3 persisted graphs.
+ * - If using checkpointing and the checkpoint interval has been reached,
+ * - Checkpoint the new graph, and put in a queue of checkpointed graphs.
+ * - Remove older checkpoints.
+ *
+ * WARNINGS:
+ * - This class should NOT be copied (since copies may conflict on which Graphs should be
+ * checkpointed).
+ * - This class removes checkpoint files once later graphs have been checkpointed.
+ * However, references to the older graphs will still return isCheckpointed = true.
+ *
+ * Example usage:
+ * val (graph1, graph2, graph3, ...) = ...
+ * val cp = new PeriodicGraphCheckpointer(graph, dir, 2)
+ * // persisted: graph1
+ * cp.updateGraph(graph2)
+ * // persisted: graph1, graph2
+ * // checkpointed: graph2
+ * cp.updateGraph(graph3)
+ * // persisted: graph1, graph2, graph3
+ * // checkpointed: graph2
+ * cp.updateGraph(graph4)
+ * // persisted: graph2, graph3, graph4
+ * // checkpointed: graph4
+ * cp.updateGraph(graph5)
+ * // persisted: graph3, graph4, graph5
+ * // checkpointed: graph4
+ *
+ * @param currentGraph Initial graph
+ * @param checkpointDir The directory for storing checkpoint files
+ * @param checkpointInterval Graphs will be checkpointed at this interval
+ * @tparam VD Vertex descriptor type
+ * @tparam ED Edge descriptor type
+ */
+private[mllib] class PeriodicGraphCheckpointer[VD, ED](
--- End diff --
I agree; almost the same code should work (except it's not clear if Graph should have exactly the same behavior since it has 2 RDDs). It's also unclear if all use cases will want the same intervals between persisting/unpersisting. I'll make a TODO and a JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72602436
@mengxr I believe that's it. Thanks for the feedback!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69971147
[Test build #25558 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25558/consoleFull) for PR 4047 at commit [`c6e4308`](https://github.com/apache/spark/commit/c6e430867ca32ca6f409f953a2d47dd04a1e6e53).
* This patch **does not merge cleanly**.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69989155
[Test build #25560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25560/consoleFull) for PR 4047 at commit [`984c414`](https://github.com/apache/spark/commit/984c414ce2bfc14fc1bef35adfca78db4770ff37).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72275188
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26421/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72295124
[Test build #26438 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26438/consoleFull) for PR 4047 at commit [`9f465b9`](https://github.com/apache/spark/commit/9f465b97b92f79c34776e0ae8f3a95b02fedaa3f).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Tokenizer(sc: SparkContext, stopwordFile: String) extends Serializable `
* ` class EMOptimizer(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72248428
[Test build #26414 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26414/consoleFull) for PR 4047 at commit [`3e0c894`](https://github.com/apache/spark/commit/3e0c8945640523f0747e2145ce5ca7b1d405b4ab).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Tokenizer(sc: SparkContext, stopwordFile: String) extends Serializable `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70146798
@EntilZha Here’s a sketch of my plan.
Datasets:
* UCI ML Repository data (also used by Asuncion et al., 2009):
* KOS
* NIPS
* NYTimes
* PubMed (full)
* Wikipedia?
Data preparation:
* Converting to bags of words:
* UCI datasets are given as word counts already.
* Wikipedia dump is text.
* I use the SimpleTokenizer in the LDAExample, which sets term = word and only accepts alphabetic characters.
* Use stopwords from @dlwh located at [https://github.com/dlwh/spark/feature/lda]
* No stemming
* Choosing vocab: For various vocabSize settings, I took the most common vocabSize terms.
Scaling tests: *(doing these first)*
* corpus size
* vocabSize
* k
* numIterations
Accuracy tests: *(doing these second)*
* train on full datasets
* Tune hyperparameters via grid search, following Asuncion et al. (2009) section 4.1.
* Can hopefully compare with their results in Fig. 5.
These tests will run on a 16-node EC2 cluster of r3.2xlarge instances.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980385
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
+ * - Persist new graph (if not yet persisted), and put in queue of persisted graphs.
+ * - Unpersist graphs from queue until there are at most 3 persisted graphs.
+ * - If using checkpointing and the checkpoint interval has been reached,
+ * - Checkpoint the new graph, and put in a queue of checkpointed graphs.
+ * - Remove older checkpoints.
+ *
+ * WARNINGS:
+ * - This class should NOT be copied (since copies may conflict on which Graphs should be
+ * checkpointed).
+ * - This class removes checkpoint files once later graphs have been checkpointed.
+ * However, references to the older graphs will still return isCheckpointed = true.
+ *
+ * Example usage:
+ * val (graph1, graph2, graph3, ...) = ...
+ * val cp = new PeriodicGraphCheckpointer(graph, dir, 2)
+ * // persisted: graph1
+ * cp.updateGraph(graph2)
+ * // persisted: graph1, graph2
+ * // checkpointed: graph2
+ * cp.updateGraph(graph3)
+ * // persisted: graph1, graph2, graph3
+ * // checkpointed: graph2
+ * cp.updateGraph(graph4)
+ * // persisted: graph2, graph3, graph4
+ * // checkpointed: graph4
+ * cp.updateGraph(graph5)
+ * // persisted: graph3, graph4, graph5
+ * // checkpointed: graph4
+ *
+ * @param currentGraph Initial graph
+ * @param checkpointDir The directory for storing checkpoint files
+ * @param checkpointInterval Graphs will be checkpointed at this interval
+ * @tparam VD Vertex descriptor type
+ * @tparam ED Edge descriptor type
+ */
+private[mllib] class PeriodicGraphCheckpointer[VD, ED](
--- End diff --
This should work on normal RDDs as well. Let's put a TODO here. Otherwise, this seems to be a little strange to live under `mllib`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72133540
[Test build #26353 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26353/consoleFull) for PR 4047 at commit [`a172ef4`](https://github.com/apache/spark/commit/a172ef4fa50745a33200c206ce3106a5b03cd2c3).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70584023
Just finished refactoring, here is the combined API/LDAModel code + Gibbs implementing it. It should give a pretty good idea of what I was thinking about using LearningState. Note though: 1. although it compiles, I haven't run it since there are some methods I still need to implement, 2. I need to update the docs after I finish implementing missing methods.
https://github.com/EntilZha/spark/tree/LDA-Refactor/mllib/src/main/scala/org/apache/spark/mllib/topicmodeling
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23729984
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
--- End diff --
Sure, that sounds good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501018
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
Is there a reason to have this as a Double rather than an Int?
+ Makes normalizing easier.
- Uncertain what signals what these counts mean (aka, whether its in a normalized state or not)
- Affects the Gibbs implementation since its part of the LDA API, which currently uses Array[Int].
Computationally, normalizing should cost about as much as returning a new array from an array (or vector) of integer counts. Thoughts?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23829150
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,473 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
+
+ def setCheckpointDir(checkpointDir: String): this.type = {
+ this.checkpointDir = Some(checkpointDir)
+ this
+ }
+
+ def clearCheckpointDir(): this.type = {
+ this.checkpointDir = None
+ this
+ }
+
+ /**
+ * Period (in iterations) between checkpoints.
+ * @see [[getCheckpointDir]]
+ */
+ def getCheckpointInterval: Int = checkpointInterval
+
+ def setCheckpointInterval(checkpointInterval: Int): this.type = {
+ this.checkpointInterval = checkpointInterval
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, which are term (word) count vectors paired with IDs.
+ * The term count vectors are "bags of words" with a fixed-size vocabulary
+ * (where the vocabulary size is the length of the vector).
+ * Document IDs must be unique and >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[(Long, Vector)]): DistributedLDAModel = {
+ val state = LDA.initialState(documents, k, getDocConcentration, getTopicConcentration, seed,
+ checkpointDir, checkpointInterval)
+ var iter = 0
+ val iterationTimes = Array.fill[Double](maxIterations)(0)
+ while (iter < maxIterations) {
+ val start = System.nanoTime()
+ state.next()
+ val elapsedSeconds = (System.nanoTime() - start) / 1e9
+ iterationTimes(iter) = elapsedSeconds
+ iter += 1
+ }
+ state.graphCheckpointer.deleteAllCheckpoints()
+ new DistributedLDAModel(state, iterationTimes)
+ }
+}
+
+
+object LDA {
--- End diff --
`private[recommendation]`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980374
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
+ * - Persist new graph (if not yet persisted), and put in queue of persisted graphs.
+ * - Unpersist graphs from queue until there are at most 3 persisted graphs.
+ * - If using checkpointing and the checkpoint interval has been reached,
+ * - Checkpoint the new graph, and put in a queue of checkpointed graphs.
+ * - Remove older checkpoints.
+ *
+ * WARNINGS:
+ * - This class should NOT be copied (since copies may conflict on which Graphs should be
+ * checkpointed).
+ * - This class removes checkpoint files once later graphs have been checkpointed.
+ * However, references to the older graphs will still return isCheckpointed = true.
+ *
+ * Example usage:
+ * val (graph1, graph2, graph3, ...) = ...
--- End diff --
use `{{{ ... }}}` for code in doc
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71934508
@jkbradley, probably the more important part isn't the commit/tag itself, but agreement on the relatively short LearningState traits.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70345841
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25689/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70434374
@hhbyyh Yes, please review the design doc linked from the JIRA. There is quite a bit of functionality which will not be in this initial PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70572184
Nice, I am not too far from completing my refactoring. How would it be best to share it, open a different PR or link to it here? Eventually, will it be two PRs... not sure best way to do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72584666
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26577/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72728215
@EntilZha @mengxr + everyone else: Thanks very much for all of the feedback!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980332
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
+
+import org.apache.spark.{SparkContext, SparkConf}
+import org.apache.spark.mllib.clustering.LDA
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * An example Latent Dirichlet Allocation (LDA) app. Run with
+ * {{{
+ * ./bin/run-example mllib.LDAExample [options] <input>
+ * }}}
+ * If you use it as a template to create your own app, please use `spark-submit` to submit your app.
+ */
+object LDAExample {
+
+ private case class Params(
+ input: Seq[String] = Seq.empty,
+ k: Int = 20,
+ maxIterations: Int = 10,
+ docConcentration: Double = -1,
+ topicConcentration: Double = -1,
+ vocabSize: Int = 10000,
+ stopwordFile: String = "",
+ checkpointDir: Option[String] = None,
+ checkpointInterval: Int = 10) extends AbstractParams[Params]
+
+ def main(args: Array[String]) {
+ val defaultParams = Params()
+
+ val parser = new OptionParser[Params]("LDAExample") {
+ head("LDAExample: an example LDA app for plain text data.")
+ opt[Int]("k")
+ .text(s"number of topics. default: ${defaultParams.k}")
+ .action((x, c) => c.copy(k = x))
+ opt[Int]("maxIterations")
+ .text(s"number of iterations of learning. default: ${defaultParams.maxIterations}")
+ .action((x, c) => c.copy(maxIterations = x))
+ opt[Double]("docConcentration")
+ .text(s"amount of topic smoothing to use (> 1.0) (-1=auto)." +
+ s" default: ${defaultParams.docConcentration}")
+ .action((x, c) => c.copy(docConcentration = x))
+ opt[Double]("topicConcentration")
+ .text(s"amount of term (word) smoothing to use (> 1.0) (-1=auto)." +
+ s" default: ${defaultParams.topicConcentration}")
+ .action((x, c) => c.copy(topicConcentration = x))
+ opt[Int]("vocabSize")
+ .text(s"number of distinct word types to use, chosen by frequency. (-1=all)" +
+ s" default: ${defaultParams.vocabSize}")
+ .action((x, c) => c.copy(vocabSize = x))
+ opt[String]("stopwordFile")
+ .text(s"filepath for a list of stopwords. Note: This must fit on a single machine." +
+ s" default: ${defaultParams.stopwordFile}")
+ .action((x, c) => c.copy(stopwordFile = x))
+ opt[String]("checkpointDir")
+ .text(s"Directory for checkpointing intermediate results." +
+ s" Checkpointing helps with recovery and eliminates temporary shuffle files on disk." +
+ s" default: ${defaultParams.checkpointDir}")
+ .action((x, c) => c.copy(checkpointDir = Some(x)))
+ opt[Int]("checkpointInterval")
+ .text(s"Iterations between each checkpoint. Only used if checkpointDir is set." +
+ s" default: ${defaultParams.checkpointInterval}")
+ .action((x, c) => c.copy(checkpointInterval = x))
+ arg[String]("<input>...")
+ .text("input paths (directories) to plain text corpora." +
+ " Each text file line should hold 1 document.")
+ .unbounded()
+ .required()
+ .action((x, c) => c.copy(input = c.input :+ x))
+ }
+
+ parser.parse(args, defaultParams).map { params =>
+ run(params)
+ }.getOrElse {
+ parser.showUsageAsError
+ sys.exit(1)
+ }
+ }
+
+ private def run(params: Params) {
+ val conf = new SparkConf().setAppName(s"LDAExample with $params")
+ val sc = new SparkContext(conf)
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // Load documents, and prepare them for LDA.
+ val preprocessStart = System.nanoTime()
+ val (corpus, vocabArray, actualNumTokens) =
+ preprocess(sc, params.input, params.vocabSize, params.stopwordFile)
+ corpus.cache()
+ val actualCorpusSize = corpus.count()
+ val actualVocabSize = vocabArray.size
+ val preprocessElapsed = (System.nanoTime() - preprocessStart) / 1e9
+
+ println()
+ println(s"Corpus summary:")
+ println(s"\t Training set size: $actualCorpusSize documents")
+ println(s"\t Vocabulary size: $actualVocabSize terms")
+ println(s"\t Training set size: $actualNumTokens tokens")
+ println(s"\t Preprocessing time: $preprocessElapsed sec")
+ println()
+
+ // Run LDA.
+ val lda = new LDA()
+ lda.setK(params.k)
+ .setMaxIterations(params.maxIterations)
+ .setDocConcentration(params.docConcentration)
+ .setTopicConcentration(params.topicConcentration)
+ .setCheckpointInterval(params.checkpointInterval)
+ if (params.checkpointDir.nonEmpty) {
+ lda.setCheckpointDir(params.checkpointDir.get)
+ }
+ val startTime = System.nanoTime()
+ val ldaModel = lda.run(corpus)
+ val elapsed = (System.nanoTime() - startTime) / 1e9
+
+ println(s"Finished training LDA model. Summary:")
+ println(s"\t Training time: $elapsed sec")
+ val avgLogLikelihood = ldaModel.logLikelihood / actualCorpusSize.toDouble
+ println(s"\t Training data average log likelihood: $avgLogLikelihood")
+ println()
+
+ // Print the topics, showing the top-weighted terms for each topic.
+ val topicIndices = ldaModel.describeTopics(maxTermsPerTopic = 10)
+ val topics = topicIndices.map { case (terms, termWeights) =>
+ terms.zip(termWeights).map { case (term, weight) => (vocabArray(term.toInt), weight) }
+ }
+ println(s"${params.k} topics:")
+ topics.zipWithIndex.foreach { case (topic, i) =>
+ println(s"TOPIC $i")
+ topic.foreach { case (term, weight) =>
+ println(s"$term\t$weight")
+ }
+ println()
+ }
+
+ }
+
+ /**
+ * Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
+ * @return (corpus, vocabulary as array, total token count in corpus)
+ */
+ private def preprocess(
+ sc: SparkContext,
+ paths: Seq[String],
+ vocabSize: Int,
+ stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+
+ // Get dataset of document texts
+ // One document per line in each text file.
+ val files: Seq[RDD[String]] = for (p <- paths) yield {
+ sc.textFile(p)
--- End diff --
`sc.textFile(paths.mkString(","))` should work, which reduces the length of lineage.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70541856
@EntilZha Thanks for taking a look! W.r.t. the class names, I really hope we can keep a single LDA class and have an optimizer parameter which lets users specify the optimization/inference method. We are trying to move away from the AlgWithOptimizer naming conventions. That will end up affecting the abstractions you were discussing too (though maybe those abstractions will exist but be private/protected).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23829160
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize, sum => brzSum}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx.{VertexId, EdgeContext, Graph}
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, Int)]]
--- End diff --
Just realized that this is not Java-friendly. In the JavaDoc, it shows up as
~~~
scala.Tuple2<Object,Object>[][] describeTopics(int maxTermsPerTopic)
~~~
If we change the return type to `Array[Array[Int], Array[Double]]`, Java would be able to recognize it correctly:
~~~
scala.Tuple2<int[],double[]>[] describeTopics(int maxTermsPerTopic)
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70029618
Mind posting the data set size (vocab, doc, etc) and type of cluster? About to start some performance tests and would be cool to hit both the Chinese dataset size and what you are testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by akopich <gi...@git.apache.org>.
Github user akopich commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501548
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
http://link.springer.com/chapter/10.1007%2F978-3-319-12580-0_3
Theorem 1. It's formulated for an arbitrary regularization. Dirichlet prior is just a special case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23724967
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
--- End diff --
Why `String` instead of an `Int`? It is not clear from the doc how the string is constructed. I'm thinking about using `Array[Vector]` as the return type and use sparse vectors to represent the distribution with thresholding. One minor downside is that the term indices are not ordered by term weights.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23726287
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
+
+ private[clustering] type TokenCount = Double
+
+ /** Term vertex IDs are {-1, -2, ..., -vocabSize} */
+ private[clustering] def term2index(term: Int): Long = -(1 + term.toLong)
+
+ private[clustering] def index2term(termIndex: Long): Int = -(1 + termIndex).toInt
+
+ private[clustering] def isDocumentVertex(v: (VertexId, _)): Boolean = v._1 >= 0
+
+ private[clustering] def isTermVertex(v: (VertexId, _)): Boolean = v._1 < 0
+
+ /**
+ * State for EM algorithm: data + parameter graph, plus algorithm parameters.
+ *
+ * @param graph EM graph, storing current parameter estimates in vertex descriptors and
+ * data (token counts) in edge descriptors.
+ * @param k Number of topics
+ * @param vocabSize Number of unique terms
+ * @param topicSmoothing "alpha"
+ * @param termSmoothing "eta"
+ */
+ private[clustering] case class LearningState(
+ graph: Graph[TopicCounts, TokenCount],
+ k: Int,
+ vocabSize: Int,
+ topicSmoothing: Double,
+ termSmoothing: Double) {
+ // TODO: Checkpoint periodically?
+ def next(): LearningState = copy(graph = step(graph))
+
+ private def step(graph: Graph[TopicCounts, TokenCount]): Graph[TopicCounts, TokenCount] = {
+ val eta = termSmoothing
+ val W = vocabSize
+ val alpha = topicSmoothing
+
+ val N_k = collectTopicTotals()
+ val sendMsg: EdgeContext[TopicCounts, TokenCount, (Boolean, TopicCounts)] => Unit =
+ (edgeContext) => {
+ // Compute N_{wj} gamma_{wjk}
+ val N_wj = edgeContext.attr
+ // E-STEP: Compute gamma_{wjk} (smoothed topic distributions), scaled by token count
+ // N_{wj}.
+ val scaledTopicDistribution: TopicCounts =
+ computePTopic(edgeContext.srcAttr, edgeContext.dstAttr, N_k, W, eta, alpha) *= N_wj
+ edgeContext.sendToDst((false, scaledTopicDistribution))
+ edgeContext.sendToSrc((false, scaledTopicDistribution))
+ }
+ // This is a hack to detect whether we could modify the values in-place.
+ // TODO: Add zero/seqOp/combOp option to aggregateMessages.
+ val mergeMsg: ((Boolean, TopicCounts), (Boolean, TopicCounts)) => (Boolean, TopicCounts) =
+ (m0, m1) => {
+ val sum =
+ if (m0._1) {
+ m0._2 += m1._2
+ } else if (m1._1) {
+ m1._2 += m0._2
+ } else {
+ m0._2 + m1._2
+ }
+ (true, sum)
+ }
+ // M-STEP: Aggregation computes new N_{kj}, N_{wk} counts.
+ val docTopicDistributions: VertexRDD[TopicCounts] =
+ graph.aggregateMessages[(Boolean, TopicCounts)](sendMsg, mergeMsg)
+ .mapValues(_._2)
+ // Update the vertex descriptors with the new counts.
+ graph.outerJoinVertices(docTopicDistributions) { (vid, oldDist, newDist) => newDist.get }
+ }
+
+ def collectTopicTotals(): TopicCounts = {
+ val numTopics = k
+ graph.vertices.filter(isTermVertex).values.fold(BDV.zeros[Double](numTopics))(_ += _)
+ }
+
+ /**
+ * Compute the log likelihood of the observed tokens, given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior; for that, use [[logPrior]].
+ * - Even with [[logPrior]], this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ */
+ lazy val logLikelihood: Double = {
+ val eta = termSmoothing
+ val alpha = topicSmoothing
+ assert(eta > 1.0)
--- End diff --
`termSmoothing` could be in `(0, 1)` given the current precondition checks. In the example code, the default is 0.1, which leads to assert error here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70572973
Sounds good! I'd recommend linking to it from here for now and creating a PR later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72348800
*Update on tests*
Summary:
* On a small dataset (20 newsgroups), it seems to work fine (on my laptop).
* On a big dataset (Wikipedia dump with close to 1 billion tokens), it's been hard to get it to run for more than 10 or 20 iterations (on a 16-node EC2 cluster).
Details:
Small dataset: You can see the output here: [https://github.com/jkbradley/spark/blob/lda-tmp/20news.lda.out]. The log likelihood improves with each iteration, and iteration running times stay about the same throughout training. The topics are really nicely divided among the newsgroups. (But I did run this using 20 topics.) I used 100 iterations and the stopwords mentioned above.
Large dataset: Even with checkpointing, it has been hard to run for many iterations, mainly because of shuffle files and checkpoint files building up. I need to spend some more time running tests. Currently, the results on the Wikipedia dump do not look good; topics are pretty much all the same. It is unclear if this is because of poor convergence, a need for parameter tuning, a need for supporting sparsity as mentioned above (which might help to force topics to differentiate), or a need for better initialization (since EM can have lots of trouble with LDA's many local minima).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by witgo <gi...@git.apache.org>.
Github user witgo commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70030602
There are two questions.
1. How to cover the long tail topic features
2. The algorithm performance in 10K -100K topic
Here is a relevant industrial practice:
"[Peacock: Learning Long-Tail Topic Features for Industrial Applications](http://arxiv.org/abs/1405.4402)"
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72140305
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26353/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980413
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
--- End diff --
Same here. I feel the doc should be in setters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501086
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
--- End diff --
This is what the run method might look like with the LearningState trait etc. I added description on what it is running from.
```scala
/**
* Learn an LDA model using the given dataset.
*
* @param documents RDD of documents, where each document is represented as a vector of term
* counts plus an ID. Document IDs must be >= 0.
* @return Inferred LDA model
*/
def runFromBagOfWords(documents: RDD[Document]): DistributedLDAModel = {
val topicSmoothing = if (this.topicSmoothing > 0) {
this.topicSmoothing
} else {
(50.0 / k) + 1.0
}
val termSmoothing = if (this.termSmoothing > 0) {
this.termSmoothing
} else {
1.1
}
var state = algorithm match {
case LearningAlgorithm.Gibbs => LDA.GibbsLearningState.initialStateFromBagOfWords(documents, k, topicSmoothing, termSmoothing, seed)
case LearningAlgorithm.EM => LDA.GibbsLearningState.initialStateFromBagOfWords(documents, k, topicSmoothing, termSmoothing, seed)
}
var iter = 0
while (iter < maxIterations) {
state = state.next()
iter += 1
}
new DistributedLDAModel(state)
}
def runFromEdges(edges: RDD[(WordId, DocId)]): DistributedLDAModel = {
val topicSmoothing = if (this.topicSmoothing > 0) {
this.topicSmoothing
} else {
(50.0 / k) + 1.0
}
val termSmoothing = if (this.termSmoothing > 0) {
this.termSmoothing
} else {
1.1
}
var state = algorithm match {
case LearningAlgorithm.Gibbs => LDA.GibbsLearningState.initialStateFromEdges(edges, k, topicSmoothing, termSmoothing, seed)
case LearningAlgorithm.EM => LDA.GibbsLearningState.initialStateFromEdges(edges, k, topicSmoothing, termSmoothing, seed)
}
var iter = 0
while (iter < maxIterations) {
state = state.next()
iter += 1
}
new DistributedLDAModel(state)
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23729365
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
--- End diff --
Okay. Let's keep this PR simple. We can add features in later PRs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72265171
@mengxr Thanks! I updated describeTopics and made a Java test suite, and made object LDA private[clustering].
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69971352
[Test build #25558 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25558/consoleFull) for PR 4047 at commit [`c6e4308`](https://github.com/apache/spark/commit/c6e430867ca32ca6f409f953a2d47dd04a1e6e53).
* This patch **fails Scala style tests**.
* This patch **does not merge cleanly**.
* This patch adds the following public classes _(experimental)_:
* ` case class Document(counts: Vector, id: Long)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501109
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * WARNING: If vocabSize and k are large, this can return a large object!
+ *
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(): Array[Array[(Double, String)]] = describeTopics(vocabSize)
+
+ /* TODO
+ * Compute the log likelihood of the observed tokens, given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior.
+ * - Even with the prior, this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated log likelihood of the data under this model
+ */
+ // def logLikelihood(documents: RDD[Document]): Double
+
+ /* TODO
+ * Compute the estimated topic distribution for each document.
+ * This is often called “theta” in the literature.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated topic distribution for each document.
+ * The returned RDD may be zipped with the given RDD, where each returned vector
+ * is a multinomial distribution over topics.
+ */
+ // def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)]
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Local LDA model.
+ * This model stores only the inferred topics.
+ * It may be used for computing topics for new documents, but it may give less accurate answers
+ * than the [[DistributedLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ *
+ * @param topics Inferred topics (vocabSize x k matrix).
+ */
+@DeveloperApi
+class LocalLDAModel private[clustering] (
+ private val topics: Matrix) extends LDAModel with Serializable {
+
+ import LDA._
+
+ override def k: Int = topics.numCols
+
+ override def vocabSize: Int = topics.numRows
+
+ override def topicsMatrix: Matrix = topics
+
+ override def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]] = {
+ val brzTopics = topics.toBreeze.toDenseMatrix
+ val topicSummary = Range(0, k).map { topicIndex =>
+ val topic = normalize(brzTopics(::, topicIndex), 1.0)
+ topic.toArray.zipWithIndex.sortBy(-_._1).take(maxTermsPerTopic)
+ }.toArray
+ topicSummary.map { topic =>
+ topic.map { case (weight, term) => (weight, term.toString) }
+ }
+ }
+
+ // TODO
+ // override def logLikelihood(documents: RDD[Document]): Double = ???
+
+ // TODO:
+ // override def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)] = ???
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Distributed LDA model.
+ * This model stores the inferred topics, the full training dataset, and the topic distributions.
+ * When computing topics for new documents, it may give more accurate answers
+ * than the [[LocalLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class DistributedLDAModel private[clustering] (
+ private val state: LDA.LearningState) extends LDAModel {
+
+ import LDA._
+
+ def toLocal: LocalLDAModel = new LocalLDAModel(topicsMatrix)
+
+ override def k: Int = state.k
+
+ override def vocabSize: Int = state.vocabSize
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ *
+ * WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.
+ */
+ override lazy val topicsMatrix: Matrix = {
+ // Collect row-major topics
+ val termTopicCounts: Array[(Int, TopicCounts)] =
+ state.graph.vertices.filter(_._1 < 0).map { case (termIndex, cnts) =>
+ (index2term(termIndex), cnts)
+ }.collect()
+ // Convert to Matrix
+ val brzTopics = BDM.zeros[Double](vocabSize, k)
+ termTopicCounts.foreach { case (term, cnts) =>
+ var j = 0
+ while (j < k) {
+ brzTopics(term, j) = cnts(j)
+ j += 1
+ }
+ }
+ Matrices.fromBreeze(brzTopics)
+ }
+
+ override def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]] = {
+ val numTopics = k
+ // Note: N_k is not needed to find the top terms, but it is needed to normalize weights
+ // to a distribution over terms.
+ val N_k: TopicCounts = state.collectTopicTotals()
+ val topicSummary = state.graph.vertices.filter(isTermVertex)
+ .mapPartitions { termVertices =>
+ // For this partition, collect the most common terms for each topic in queues:
+ // queues(topic) = queue of (term weight, term index).
+ // Term weights are N_{wk} / N_k.
+ val queues = Array.fill(numTopics)(new BoundedPriorityQueue[(Double, Int)](maxTermsPerTopic))
+ for ((termId, n_wk) <- termVertices) {
+ var topic = 0
+ while (topic < numTopics) {
+ queues(topic) += (n_wk(topic) / N_k(topic) -> index2term(termId.toInt))
+ topic += 1
+ }
+ }
+ Iterator(queues)
+ }.reduce { (q1, q2) =>
+ q1.zip(q2).foreach { case (a, b) => a ++= b}
+ q1
+ }.map(_.toArray.sortBy(-_._1))
+ topicSummary.map { topic =>
+ topic.map { case (weight, term) => (weight, term.toString) }
+ }
+ }
+
+ // TODO
+ // override def logLikelihood(documents: RDD[Document]): Double = ???
+
+ /**
+ * Compute the log likelihood of the observed tokens in the training set,
+ * given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior; for that, use [[logPrior]].
+ * - Even with [[logPrior]], this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ */
+ def logLikelihood = state.logLikelihood
+
+ /**
+ * Compute the log probability of the current parameter estimate, under the prior:
+ * log P(topics, topic distributions for docs | alpha, eta)
+ */
+ def logPrior = state.logPrior
+
+ /**
+ * For each document in the training set, return the distribution over topics for that document
+ * (i.e., "theta_doc").
+ *
+ * @return RDD of (document ID, topic distribution) pairs
+ */
+ def topicDistributions: RDD[(Long, Vector)] = {
--- End diff --
Similarly,
```scala
state.topicDistributions()
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70345837
[Test build #25689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25689/consoleFull) for PR 4047 at commit [`648f66c`](https://github.com/apache/spark/commit/648f66cc97eefd5698480402a5cf859c618c1467).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` case class Document(counts: Vector, id: Long)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23985441
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
--- End diff --
Will do
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72573927
[Test build #26569 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26569/consoleFull) for PR 4047 at commit [`e3980d2`](https://github.com/apache/spark/commit/e3980d2967ae45d1477384a055abd1c9d7975dbd).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23724950
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
--- End diff --
Could we make `Document` private and use `RDD[(Long, Vector)]` directly in `LDA.run`? `Document` makes the implementation easier to read but I feel it adds an extra step for users to call `LDA.run`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by akopich <gi...@git.apache.org>.
Github user akopich commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501440
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
@jkbradley But why do you call it smoothing? If 0 < alpha < 1, Dirichlet prior does not smooth but sparsify.
By the way, correctness of EM algorithm for 0 < alpha < 1 was proven by Vorontsov (2014) -- it just involves a positive cut before normalizing counters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72275175
[Test build #26421 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26421/consoleFull) for PR 4047 at commit [`1db89e2`](https://github.com/apache/spark/commit/1db89e2964aee34f9c33300be271dde41d61a782).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class Tokenizer(sc: SparkContext, stopwordFile: String) extends Serializable `
* ` class EMOptimizer(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71932163
It would be very helpful if we can stabilize the LearningState API/Trait, and you point to a commit/tag for that. This would help me finish Gibbs in parallel with EM without having to refactor after this PR goes through. I am thinking that this PR should be finished first, then I will open a followup PR to merge Gibbs in (and review it). I still need to open that ticket when I get a chance.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69978003
[Test build #25560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25560/consoleFull) for PR 4047 at commit [`984c414`](https://github.com/apache/spark/commit/984c414ce2bfc14fc1bef35adfca78db4770ff37).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by dlwh <gi...@git.apache.org>.
Github user dlwh commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501410
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
(i originally did it this way because Breeze doesn't support mixed-type operations, so it made a lot of things prettier to just use doubles.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501409
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
+
+ private[clustering] type TokenCount = Double
+
+ /** Term vertex IDs are {-1, -2, ..., -vocabSize} */
+ private[clustering] def term2index(term: Int): Long = -(1 + term.toLong)
+
+ private[clustering] def index2term(termIndex: Long): Int = -(1 + termIndex).toInt
+
+ private[clustering] def isDocumentVertex(v: (VertexId, _)): Boolean = v._1 >= 0
+
+ private[clustering] def isTermVertex(v: (VertexId, _)): Boolean = v._1 < 0
+
+ /**
+ * State for EM algorithm: data + parameter graph, plus algorithm parameters.
+ *
+ * @param graph EM graph, storing current parameter estimates in vertex descriptors and
+ * data (token counts) in edge descriptors.
+ * @param k Number of topics
+ * @param vocabSize Number of unique terms
+ * @param topicSmoothing "alpha"
+ * @param termSmoothing "eta"
+ */
+ private[clustering] case class LearningState(
+ graph: Graph[TopicCounts, TokenCount],
+ k: Int,
+ vocabSize: Int,
+ topicSmoothing: Double,
+ termSmoothing: Double) {
+ // TODO: Checkpoint periodically?
+ def next(): LearningState = copy(graph = step(graph))
+
+ private def step(graph: Graph[TopicCounts, TokenCount]): Graph[TopicCounts, TokenCount] = {
+ val eta = termSmoothing
+ val W = vocabSize
+ val alpha = topicSmoothing
+
+ val N_k = collectTopicTotals()
+ val sendMsg: EdgeContext[TopicCounts, TokenCount, (Boolean, TopicCounts)] => Unit =
+ (edgeContext) => {
+ // Compute N_{wj} gamma_{wjk}
+ val N_wj = edgeContext.attr
+ // E-STEP: Compute gamma_{wjk} (smoothed topic distributions), scaled by token count
+ // N_{wj}.
+ val scaledTopicDistribution: TopicCounts =
+ computePTopic(edgeContext.srcAttr, edgeContext.dstAttr, N_k, W, eta, alpha) *= N_wj
+ edgeContext.sendToDst((false, scaledTopicDistribution))
+ edgeContext.sendToSrc((false, scaledTopicDistribution))
+ }
+ // This is a hack to detect whether we could modify the values in-place.
+ // TODO: Add zero/seqOp/combOp option to aggregateMessages.
+ val mergeMsg: ((Boolean, TopicCounts), (Boolean, TopicCounts)) => (Boolean, TopicCounts) =
+ (m0, m1) => {
+ val sum =
+ if (m0._1) {
+ m0._2 += m1._2
+ } else if (m1._1) {
+ m1._2 += m0._2
+ } else {
+ m0._2 + m1._2
+ }
+ (true, sum)
+ }
+ // M-STEP: Aggregation computes new N_{kj}, N_{wk} counts.
+ val docTopicDistributions: VertexRDD[TopicCounts] =
+ graph.aggregateMessages[(Boolean, TopicCounts)](sendMsg, mergeMsg)
+ .mapValues(_._2)
+ // Update the vertex descriptors with the new counts.
+ graph.outerJoinVertices(docTopicDistributions) { (vid, oldDist, newDist) => newDist.get }
+ }
+
+ def collectTopicTotals(): TopicCounts = {
--- End diff --
Sure, I'll do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69979680
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25559/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23729109
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
--- End diff --
That adds some complexity to the code. We can say `id` must be nonnegative and unique in the doc, and check the signs while constructing the graph. Later if we can handle arbitrary ids, we can remove that restriction, which is still backward compatible.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72248302
[Test build #26414 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26414/consoleFull) for PR 4047 at commit [`3e0c894`](https://github.com/apache/spark/commit/3e0c8945640523f0747e2145ce5ca7b1d405b4ab).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23982107
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
--- End diff --
What would you vote for?
* PeriodicGraphCheckpointer calls count() on vertices and edges
* PeriodicGraphCheckpointer stays as is, where it tries to remove checkpoints but does not fail if they do not exist.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72559262
*Another test update*
Large dataset: It's still unclear to me why I cannot run for more than 10 or 15 iterations on the big Wikipedia dataset. Checkpointing on S3 works. Persisting is working. Checkpointing should be limiting the size of shuffle files. Sometimes executors run out of space, and sometimes connections die (despite long time-out settings). We'll need to do more testing to figure out how to scale this up more, but I think it's good enough for medium-size datasets.
I'm doing some clean-ups and making a test suite for the checkpointer utility.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72295132
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26438/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23727601
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
+
+ private[clustering] type TokenCount = Double
+
+ /** Term vertex IDs are {-1, -2, ..., -vocabSize} */
+ private[clustering] def term2index(term: Int): Long = -(1 + term.toLong)
+
+ private[clustering] def index2term(termIndex: Long): Int = -(1 + termIndex).toInt
+
+ private[clustering] def isDocumentVertex(v: (VertexId, _)): Boolean = v._1 >= 0
+
+ private[clustering] def isTermVertex(v: (VertexId, _)): Boolean = v._1 < 0
+
+ /**
+ * State for EM algorithm: data + parameter graph, plus algorithm parameters.
+ *
+ * @param graph EM graph, storing current parameter estimates in vertex descriptors and
+ * data (token counts) in edge descriptors.
+ * @param k Number of topics
+ * @param vocabSize Number of unique terms
+ * @param topicSmoothing "alpha"
+ * @param termSmoothing "eta"
+ */
+ private[clustering] case class LearningState(
+ graph: Graph[TopicCounts, TokenCount],
+ k: Int,
+ vocabSize: Int,
+ topicSmoothing: Double,
+ termSmoothing: Double) {
+ // TODO: Checkpoint periodically?
+ def next(): LearningState = copy(graph = step(graph))
+
+ private def step(graph: Graph[TopicCounts, TokenCount]): Graph[TopicCounts, TokenCount] = {
+ val eta = termSmoothing
+ val W = vocabSize
+ val alpha = topicSmoothing
+
+ val N_k = collectTopicTotals()
+ val sendMsg: EdgeContext[TopicCounts, TokenCount, (Boolean, TopicCounts)] => Unit =
+ (edgeContext) => {
+ // Compute N_{wj} gamma_{wjk}
+ val N_wj = edgeContext.attr
+ // E-STEP: Compute gamma_{wjk} (smoothed topic distributions), scaled by token count
+ // N_{wj}.
+ val scaledTopicDistribution: TopicCounts =
+ computePTopic(edgeContext.srcAttr, edgeContext.dstAttr, N_k, W, eta, alpha) *= N_wj
+ edgeContext.sendToDst((false, scaledTopicDistribution))
+ edgeContext.sendToSrc((false, scaledTopicDistribution))
+ }
+ // This is a hack to detect whether we could modify the values in-place.
+ // TODO: Add zero/seqOp/combOp option to aggregateMessages.
+ val mergeMsg: ((Boolean, TopicCounts), (Boolean, TopicCounts)) => (Boolean, TopicCounts) =
+ (m0, m1) => {
+ val sum =
+ if (m0._1) {
+ m0._2 += m1._2
+ } else if (m1._1) {
+ m1._2 += m0._2
+ } else {
+ m0._2 + m1._2
+ }
+ (true, sum)
+ }
+ // M-STEP: Aggregation computes new N_{kj}, N_{wk} counts.
+ val docTopicDistributions: VertexRDD[TopicCounts] =
+ graph.aggregateMessages[(Boolean, TopicCounts)](sendMsg, mergeMsg)
+ .mapValues(_._2)
+ // Update the vertex descriptors with the new counts.
+ graph.outerJoinVertices(docTopicDistributions) { (vid, oldDist, newDist) => newDist.get }
+ }
+
+ def collectTopicTotals(): TopicCounts = {
+ val numTopics = k
+ graph.vertices.filter(isTermVertex).values.fold(BDV.zeros[Double](numTopics))(_ += _)
+ }
+
+ /**
+ * Compute the log likelihood of the observed tokens, given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior; for that, use [[logPrior]].
+ * - Even with [[logPrior]], this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ */
+ lazy val logLikelihood: Double = {
+ val eta = termSmoothing
+ val alpha = topicSmoothing
+ assert(eta > 1.0)
--- End diff --
Yep, my testing branch has diverged here. I'm getting it stable before I merge it back into this branch.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501438
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
Is there a reason to use Ints at all? I worry about overflow for Int. Long might be OK, but I'd vote for Double for simplicity.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72578498
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26566/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23500980
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
Is it worth aliasing topicSmoothing/termSmoothing to alpha/beta? Might make it clearer/easier to use.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980371
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
--- End diff --
Should users be responsible for materializing graphs? If users doesn't materialize graphs, then `removeCheckpointFile` will happen before the RDDs get checkpointed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72608930
[Test build #26624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26624/consoleFull) for PR 4047 at commit [`77e8814`](https://github.com/apache/spark/commit/77e8814073c10ee8f8753eb68412b1b86450af31).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class EMOptimizer(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72595022
[Test build #574 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/574/consoleFull) for PR 4047 at commit [`589728b`](https://github.com/apache/spark/commit/589728b436b9173c880654fe97a5a9640aee639b).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71146326
Thoughts on the API @jkbradley? I am working in implementing those last few functions and performance tests. I am aiming to have the Gibbs version ready (if nothing goes wrong) by the end of the weekend.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72601749
[Test build #574 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/574/consoleFull) for PR 4047 at commit [`589728b`](https://github.com/apache/spark/commit/589728b436b9173c880654fe97a5a9640aee639b).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-69989172
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25560/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23728228
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
--- End diff --
Items which will be important to get in are:
* Tokenizer (for spark.mllib)
* the API to work with plain text
* prediction (on new documents) as you're mentioning
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980335
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
+
+import org.apache.spark.{SparkContext, SparkConf}
+import org.apache.spark.mllib.clustering.LDA
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * An example Latent Dirichlet Allocation (LDA) app. Run with
+ * {{{
+ * ./bin/run-example mllib.LDAExample [options] <input>
+ * }}}
+ * If you use it as a template to create your own app, please use `spark-submit` to submit your app.
+ */
+object LDAExample {
+
+ private case class Params(
+ input: Seq[String] = Seq.empty,
+ k: Int = 20,
+ maxIterations: Int = 10,
+ docConcentration: Double = -1,
+ topicConcentration: Double = -1,
+ vocabSize: Int = 10000,
+ stopwordFile: String = "",
+ checkpointDir: Option[String] = None,
+ checkpointInterval: Int = 10) extends AbstractParams[Params]
+
+ def main(args: Array[String]) {
+ val defaultParams = Params()
+
+ val parser = new OptionParser[Params]("LDAExample") {
+ head("LDAExample: an example LDA app for plain text data.")
+ opt[Int]("k")
+ .text(s"number of topics. default: ${defaultParams.k}")
+ .action((x, c) => c.copy(k = x))
+ opt[Int]("maxIterations")
+ .text(s"number of iterations of learning. default: ${defaultParams.maxIterations}")
+ .action((x, c) => c.copy(maxIterations = x))
+ opt[Double]("docConcentration")
+ .text(s"amount of topic smoothing to use (> 1.0) (-1=auto)." +
+ s" default: ${defaultParams.docConcentration}")
+ .action((x, c) => c.copy(docConcentration = x))
+ opt[Double]("topicConcentration")
+ .text(s"amount of term (word) smoothing to use (> 1.0) (-1=auto)." +
+ s" default: ${defaultParams.topicConcentration}")
+ .action((x, c) => c.copy(topicConcentration = x))
+ opt[Int]("vocabSize")
+ .text(s"number of distinct word types to use, chosen by frequency. (-1=all)" +
+ s" default: ${defaultParams.vocabSize}")
+ .action((x, c) => c.copy(vocabSize = x))
+ opt[String]("stopwordFile")
+ .text(s"filepath for a list of stopwords. Note: This must fit on a single machine." +
+ s" default: ${defaultParams.stopwordFile}")
+ .action((x, c) => c.copy(stopwordFile = x))
+ opt[String]("checkpointDir")
+ .text(s"Directory for checkpointing intermediate results." +
+ s" Checkpointing helps with recovery and eliminates temporary shuffle files on disk." +
+ s" default: ${defaultParams.checkpointDir}")
+ .action((x, c) => c.copy(checkpointDir = Some(x)))
+ opt[Int]("checkpointInterval")
+ .text(s"Iterations between each checkpoint. Only used if checkpointDir is set." +
+ s" default: ${defaultParams.checkpointInterval}")
+ .action((x, c) => c.copy(checkpointInterval = x))
+ arg[String]("<input>...")
+ .text("input paths (directories) to plain text corpora." +
+ " Each text file line should hold 1 document.")
+ .unbounded()
+ .required()
+ .action((x, c) => c.copy(input = c.input :+ x))
+ }
+
+ parser.parse(args, defaultParams).map { params =>
+ run(params)
+ }.getOrElse {
+ parser.showUsageAsError
+ sys.exit(1)
+ }
+ }
+
+ private def run(params: Params) {
+ val conf = new SparkConf().setAppName(s"LDAExample with $params")
+ val sc = new SparkContext(conf)
+
+ Logger.getRootLogger.setLevel(Level.WARN)
+
+ // Load documents, and prepare them for LDA.
+ val preprocessStart = System.nanoTime()
+ val (corpus, vocabArray, actualNumTokens) =
+ preprocess(sc, params.input, params.vocabSize, params.stopwordFile)
+ corpus.cache()
+ val actualCorpusSize = corpus.count()
+ val actualVocabSize = vocabArray.size
+ val preprocessElapsed = (System.nanoTime() - preprocessStart) / 1e9
+
+ println()
+ println(s"Corpus summary:")
+ println(s"\t Training set size: $actualCorpusSize documents")
+ println(s"\t Vocabulary size: $actualVocabSize terms")
+ println(s"\t Training set size: $actualNumTokens tokens")
+ println(s"\t Preprocessing time: $preprocessElapsed sec")
+ println()
+
+ // Run LDA.
+ val lda = new LDA()
+ lda.setK(params.k)
+ .setMaxIterations(params.maxIterations)
+ .setDocConcentration(params.docConcentration)
+ .setTopicConcentration(params.topicConcentration)
+ .setCheckpointInterval(params.checkpointInterval)
+ if (params.checkpointDir.nonEmpty) {
+ lda.setCheckpointDir(params.checkpointDir.get)
+ }
+ val startTime = System.nanoTime()
+ val ldaModel = lda.run(corpus)
+ val elapsed = (System.nanoTime() - startTime) / 1e9
+
+ println(s"Finished training LDA model. Summary:")
+ println(s"\t Training time: $elapsed sec")
+ val avgLogLikelihood = ldaModel.logLikelihood / actualCorpusSize.toDouble
+ println(s"\t Training data average log likelihood: $avgLogLikelihood")
+ println()
+
+ // Print the topics, showing the top-weighted terms for each topic.
+ val topicIndices = ldaModel.describeTopics(maxTermsPerTopic = 10)
+ val topics = topicIndices.map { case (terms, termWeights) =>
+ terms.zip(termWeights).map { case (term, weight) => (vocabArray(term.toInt), weight) }
+ }
+ println(s"${params.k} topics:")
+ topics.zipWithIndex.foreach { case (topic, i) =>
+ println(s"TOPIC $i")
+ topic.foreach { case (term, weight) =>
+ println(s"$term\t$weight")
+ }
+ println()
+ }
+
+ }
+
+ /**
+ * Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
+ * @return (corpus, vocabulary as array, total token count in corpus)
+ */
+ private def preprocess(
+ sc: SparkContext,
+ paths: Seq[String],
+ vocabSize: Int,
+ stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+
+ // Get dataset of document texts
+ // One document per line in each text file.
+ val files: Seq[RDD[String]] = for (p <- paths) yield {
+ sc.textFile(p)
+ }
+ val textRDD: RDD[String] = files.reduce(_ ++ _) // combine results from multiple paths
+
+ // Split text into words
+ val tokenizer = new SimpleTokenizer(sc, stopwordFile)
+ val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
+ id -> tokenizer.getWords(text)
+ }
+ tokenized.cache()
+
+ // Counts words: RDD[(word, wordCount)]
+ val wordCounts: RDD[(String, Long)] = tokenized
+ .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
+ .reduceByKey(_ + _)
+ // Sort words, and select vocab
+ val sortedWC = wordCounts.sortBy(_._2, ascending = false)
+ val selectedWC: Array[(String, Long)] = if (vocabSize == -1) {
+ sortedWC.collect()
--- End diff --
minor: count first then decide whether to sort, which is more expensive. If you do want to have the indices ordered by word frequency, note that `collect()` doesn't guarantee the ordering of results.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70146828
@witgo I agree that there are 2 different use regimes for LDA: interpretable topics and featurization. The current implementation follows pretty much every other graph-based implementation I’ve seen:
* 1 vertex per document + 1 vertex per term
* Each vertex stores a vector of length # topics.
* On each iteration, each doc vertex must communicate its vector to any connected term vertices (and likewise for term vertices), via map-reduce stages over triplets.
I have not heard of methods which can avoid this amount of communication for LDA. I’m sure the implementation can be optimized, so please make comments here or JIRAs afterwards about that. For modified models, it might be possible to communicate less: sparsity-inducing priors, hierarchical models, etc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72265197
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26419/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23985029
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
--- End diff --
I think we should update the documentation and put a note about this. Users are responsible for materializing graphs at proper time and we track the last checkpoint.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72608935
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26624/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980387
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/impl/PeriodicGraphCheckpointer.scala ---
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.impl
+
+import scala.collection.mutable
+
+import org.apache.hadoop.fs.{Path, FileSystem}
+
+import org.apache.spark.Logging
+import org.apache.spark.graphx.Graph
+import org.apache.spark.storage.StorageLevel
+
+
+/**
+ * This class helps with persisting and checkpointing Graphs.
+ *
+ * This class maintains a FIFO queue of Graphs, each of which is persisted and some of which are
+ * checkpointed. Once one Graph has been checkpointed, then previous RDDs are unpersisted and their
+ * checkpoint files are removed.
+ *
+ * Users should call [[PeriodicGraphCheckpointer.updateGraph()]] when a new graph has been created,
+ * before the graph has been materialized. When called, this does the following:
+ * - Persist new graph (if not yet persisted), and put in queue of persisted graphs.
+ * - Unpersist graphs from queue until there are at most 3 persisted graphs.
+ * - If using checkpointing and the checkpoint interval has been reached,
+ * - Checkpoint the new graph, and put in a queue of checkpointed graphs.
+ * - Remove older checkpoints.
+ *
+ * WARNINGS:
+ * - This class should NOT be copied (since copies may conflict on which Graphs should be
+ * checkpointed).
+ * - This class removes checkpoint files once later graphs have been checkpointed.
+ * However, references to the older graphs will still return isCheckpointed = true.
+ *
+ * Example usage:
+ * val (graph1, graph2, graph3, ...) = ...
+ * val cp = new PeriodicGraphCheckpointer(graph, dir, 2)
+ * // persisted: graph1
+ * cp.updateGraph(graph2)
+ * // persisted: graph1, graph2
+ * // checkpointed: graph2
+ * cp.updateGraph(graph3)
+ * // persisted: graph1, graph2, graph3
+ * // checkpointed: graph2
+ * cp.updateGraph(graph4)
+ * // persisted: graph2, graph3, graph4
+ * // checkpointed: graph4
+ * cp.updateGraph(graph5)
+ * // persisted: graph3, graph4, graph5
+ * // checkpointed: graph4
+ *
+ * @param currentGraph Initial graph
+ * @param checkpointDir The directory for storing checkpoint files
+ * @param checkpointInterval Graphs will be checkpointed at this interval
+ * @tparam VD Vertex descriptor type
+ * @tparam ED Edge descriptor type
+ */
+private[mllib] class PeriodicGraphCheckpointer[VD, ED](
+ var currentGraph: Graph[VD, ED],
+ val checkpointDir: Option[String],
+ val checkpointInterval: Int) extends Logging {
+
+ /** FIFO queue of past checkpointed RDDs */
+ private val checkpointQueue = mutable.Queue[Graph[VD, ED]]()
+
+ /** FIFO queue of past persisted RDDs */
+ private val persistedQueue = mutable.Queue[Graph[VD, ED]]()
+
+ /** Number of times [[updateGraph()]] has been called */
+ private var updateCount = 0
+
+ /**
+ * Spark Context for the Graphs given to this checkpointer.
+ * NOTE: This code assumes that only one SparkContext is used for the given graphs.
+ */
+ private val sc = currentGraph.vertices.sparkContext
+
+ // If a checkpoint directory is given, and there's no prior checkpoint directory,
+ // then set the checkpoint directory with the given one.
+ if (checkpointDir.nonEmpty && sc.getCheckpointDir.isEmpty) {
--- End diff --
See my previous comment about setting `checkpointDir`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71998995
@EntilZha The discussion was about what *private* interfaces Gibbs sampling needs. It is definitely useful, but the one really blocks this PR would be whether the public APIs proposed are compatible with Gibbs. I feel it would be easier to discuss the changes you proposed along with the Gibbs implementation in a later PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23727558
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
--- End diff --
If you look at the design doc, the idea is to be able to work with either word count vectors or Strings:
* training can take either:
* word count vectors (this PR)
* optionally, with a dictionary (later PR)
* Strings (later PR) (using a different run() method)
* to get topics:
* topicsMatrix returns topics as vectors (using term indices)
* describeTopics returns topics using strings (which are the actual terms if training was done with Strings or if a dictionary was given)
I think providing a sorted description of topics will be very helpful. Otherwise, users will almost always have to implement it on their own.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71946931
@mengxr, correct, its not public, but it would be helpful to at least get a minimal Trait which I can base my PR on (something similar to what I proposed earlier). That being said, I could include the changes in structure to Trait and Gibbs in a PR... seems easier to get it right the first time though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72083440
@EntilZha Feel free to continue the discussion of your PR in a separate JIRA though (but I agree that here is not the ideal place since this discussion will get forgotten once this PR gets merged).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23502411
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
I used Ints primarily because in the Gibbs implementation I always used integer counts and didn't use a normalized distribution anywhere except in the final output. To me, it seems like using an integer type (either Int or Long) is simpler, but I recognize maybe I am missing something. Could you reference some of those places (I am curious if its general to LDA or specific to EM that makes it easier). Secondary reason, since I always use integers, Int/Long guarantees no loss of precision, granted, loss of precision would be very rare and you would need to get very high in counts for it to happen (unlikely).
On Int vs Long, agreed if you want to manage overflow, then Long is good. The tradeoff is that for cases where a single given topic doesn;t exceed MAX_INT (which is ~2 billion), you are effectively doubling memory usage since most of the memory in LDA is consumed by Arrays (or Vectors) of topic counts.
Again though, since the data type is tied to the implementation of EM, this would very neatly fit within EMLearningState and have the methods of the LearningState trait return regular types instead of aliased types (Arrays, Ints, Longs, Doubles, Vectors...), so it may be a moot point with regards to affecting Gibbs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72600046
@jkbradley Both setter and getter are public methods and hence both of them should have JavaDoc. I prefer doc the default value in setter because it is used when a user want to change the parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980329
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
--- End diff --
organize imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72578196
(That was the previous commit failing)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501399
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
+
+ /**
+ * Vector over topics (length k) of token counts.
+ * The meaning of these counts can vary, and it may or may not be normalized to be a distribution.
+ */
+ private[clustering] type TopicCounts = BDV[Double]
--- End diff --
So that probably means that it would be good to confirm the LearningState API to make sure it has ways to return both?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501394
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
--- End diff --
Why do we need to initialize from edges? Why not just initialize from bags of words (and create the edges internally)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72133253
I just pushed a big update with the following changes:
Added checkpointing to LDA
* new class PeriodicGraphCheckpointer
* params checkpointDir, checkpointInterval to LDA
Internal changes to LDA
* Changed State to be mutable (since it needs to hold the PeriodicGraphCheckpointer)
* Added timing instrumentation
* Changed DistributedLDAModel not to hold a LearningState
* This was needed since LearningState needs to hold a PeriodicGraphCheckpointer. We should be able to copy a model, but we cannot copy PeriodicGraphCheckpointer instances.
* Add checks for valid ranges of eta, alpha
* Rename “LearningState” to “EMOptimizer”
Public changes to LDA
* Updated naming of describeTopics, and commented out version using String
* Removed Document type in favor of (Long, Vector)
* Changed doc ID restriction to be: id must be nonnegative and unique in the doc (instead of 0,1,2,...)
* Renamed params: termSmoothing -> topicConcentration, topicSmoothing -> docConcentration
* Also added aliases alpha, beta
Also, this update includes code from [https://github.com/apache/spark/pull/4253] which will be removed once that PR gets merged.
These updates fix the issue with iterations taking longer and longer.
Items remaining:
* run large test again
* remove LDATiming
* possibly port vocab computation from LDATiming (if it's faster...to be tested soon)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23981527
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -0,0 +1,285 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.mllib
+
+import scala.collection.mutable
+
+import java.text.BreakIterator
+
+import scopt.OptionParser
+
+import org.apache.log4j.{Level, Logger}
--- End diff --
https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide#SparkCodeStyleGuide-Imports
* java.* and javax.*
* scala.*
* Third-party libraries (org.*, com.*, etc)
* Project classes (org.apache.spark.*)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72601681
@mengxr OK, I'll copy the doc to both, but only put the default value in the setter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70396762
Is there plan to include the inference for new (unseen) document based on the generated distribution? Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72577270
[Test build #26565 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26565/consoleFull) for PR 4047 at commit [`e391474`](https://github.com/apache/spark/commit/e3914748187283a0ea3c929e91a0d5cae44e7b28).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` class EMOptimizer(`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23502479
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
--- End diff --
Primarily, convenience, there are some data formats where creating from an edge list is more natural. For example, one input format which I have worked with is formatted like:
```
DocId WordId NumOccurrences
```
That input format is very easy to do with edges, but requires more work to group by DocId, then perform an aggregate. In the case of receiving documents as RDD[String], I think both methods are comparable in complexity if we don't provide an interface for doing so in first release. Summary, I think they are both useful different ways to input data.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23982202
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
--- End diff --
The current behavior is actually OK, but I'll make it clearer in the LDA doc. It is passed to PeriodicGraphCheckpointer, where it is ignored if sc.checkpointDir is already set.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501523
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
I agree "smoothing" is odd if the regularization makes the solution sparser, but it's pretty standard. I guess "alpha" and "eta" are OK if we make the meaning very clear in the docs. Or we could try to follow the GLM example and say "termRegParam" and "topicRegParam."
@akopich Do you have a link for that paper? I found papers of his for LDA but haven't found one with a proof of correctness.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70024523
By the way, I'm running larger-scale tests, and I'll post results once they are ready!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23727115
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
--- End diff --
I'm OK either way, I guess. (simpler to call vs. clearer types)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72572315
[Test build #26565 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26565/consoleFull) for PR 4047 at commit [`e391474`](https://github.com/apache/spark/commit/e3914748187283a0ea3c929e91a0d5cae44e7b28).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980421
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
--- End diff --
The default value should appear in the setter doc.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23980406
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,477 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.graphx._
+import org.apache.spark.graphx.impl.GraphImpl
+import org.apache.spark.mllib.impl.PeriodicGraphCheckpointer
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var docConcentration: Double,
+ private var topicConcentration: Double,
+ private var seed: Long,
+ private var checkpointDir: Option[String],
+ private var checkpointInterval: Int) extends Logging {
+
+ def this() = this(k = 10, maxIterations = 20, docConcentration = -1, topicConcentration = -1,
+ seed = Utils.random.nextLong(), checkpointDir = None, checkpointInterval = 10)
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on each document's distribution over topics
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 1.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then docConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getDocConcentration: Double = {
+ if (this.docConcentration == -1) {
+ (50.0 / k) + 1.0
+ } else {
+ this.docConcentration
+ }
+ }
+
+ def setDocConcentration(docConcentration: Double): this.type = {
+ require(docConcentration > 1.0 || docConcentration == -1.0,
+ s"LDA docConcentration must be > 1.0 (or -1 for auto), but was set to $docConcentration")
+ this.docConcentration = docConcentration
+ this
+ }
+
+ /** Alias for [[getDocConcentration]] */
+ def getAlpha: Double = getDocConcentration
+
+ /** Alias for [[setDocConcentration()]] */
+ def setAlpha(alpha: Double): this.type = setDocConcentration(alpha)
+
+ /**
+ * Term smoothing parameter (commonly named "beta" or "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on each topic's distribution over terms
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009).
+ *
+ * This value should be > 0.0.
+ * If set to -1, then topicConcentration is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ *
+ * Note: The restriction > 1.0 may be relaxed in the future (allowing sparse solutions),
+ * but values in (0,1) are not yet supported.
+ */
+ def getTopicConcentration: Double = {
+ if (this.topicConcentration == -1) {
+ 1.1
+ } else {
+ this.topicConcentration
+ }
+ }
+
+ def setTopicConcentration(topicConcentration: Double): this.type = {
+ require(topicConcentration > 1.0 || topicConcentration == -1.0,
+ s"LDA topicConcentration must be > 1.0 (or -1 for auto), but was set to $topicConcentration")
+ this.topicConcentration = topicConcentration
+ this
+ }
+
+ /** Alias for [[getTopicConcentration]] */
+ def getBeta: Double = getTopicConcentration
+
+ /** Alias for [[setTopicConcentration()]] */
+ def setBeta(beta: Double): this.type = setBeta(beta)
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Directory for storing checkpoint files during learning.
+ * This is not necessary, but checkpointing helps with recovery (when nodes fail).
+ * It also helps with eliminating temporary shuffle files on disk, which can be important when
+ * LDA is run for many iterations.
+ */
+ def getCheckpointDir: Option[String] = checkpointDir
+
+ def setCheckpointDir(checkpointDir: String): this.type = {
+ this.checkpointDir = Some(checkpointDir)
+ this
+ }
+
+ def clearCheckpointDir(): this.type = {
+ this.checkpointDir = None
+ this
+ }
+
+ /**
+ * Period (in iterations) between checkpoints.
--- End diff --
Need to mention the default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72579236
(Lots of unrelated failures)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/4047
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72602871
[Test build #26624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26624/consoleFull) for PR 4047 at commit [`77e8814`](https://github.com/apache/spark/commit/77e8814073c10ee8f8753eb68412b1b86450af31).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72572861
[Test build #26566 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26566/consoleFull) for PR 4047 at commit [`4ae2a7d`](https://github.com/apache/spark/commit/4ae2a7df685badae3b7a3b8e94a871186c148a78).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501071
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
--- End diff --
Currently, this is what I have as the contents of object LDA, to provide something for LearningStates (Gibbs, EM), to implement.
LearningState is intended to be extended by a class which represents the model/data/etc
LearningStateInitialization is intended to be extended by the companion object to create static method hooks to create a LearningState. I tried to make a single trait and let Scala see that between the companion object/class it would recognize its satisfied, but no dice (if there is a way to do this I would love to know.
```scala
object LearningAlgorithm extends Enumeration {
type LearningAlgorithm = Value
val Gibbs, EM = Value
}
type TopicCounts = Array[Int]
case class Document(counts: Vector, id: Long)
type DocId = Long
type WordId = Long
trait LearningState {
def logPrior(): Double
def logLikelihood(): Double
def collectTopicTotals(): TopicCounts
def topicDistributions(): RDD[(Long, Vector)]
def describeTopics(maxTermsPerTopic: Long): Array[Array[(Double, String)]]
def topicsMatrix: Matrix
def next(): LearningState
def vocabSize(): Long
def k: Int
}
trait LearningStateInitialization {
def initialStateFromBagOfWords(docs: RDD[Document], k: Int, alpha: Double, beta: Double, seed: Long): LearningState
def initialStateFromEdges(edges: RDD[(WordId, DocId)], k: Int, alpha: Double, beta: Double, seed: Long): LearningState
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501379
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
--- End diff --
I like having "smoothing" in the names since that's easier to understand for people who have not read LDA papers. I agree the difference between smoothing "topics" and "terms" is hard to understand. (If it were not for the precedent set by the Stanford NLP toolbox, I would actually vote to relabel "termSmoothing" --> "topicSmoothing" and "topicSmoothing" --> "docSmoothing" since we are smoothing distributions, not smoothing elements of distributions.)
One other issue with alpha/beta is that papers are not consistent about parameter names. ("beta" is more often called "eta" in what I've read.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72577277
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26565/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72591879
@mengxr Almost done with updating per your feedback, but 1 question: Do you recommend doc only in the setter methods, or in both the getters and setters?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70542992
What might be the best way to have the EM and Gibbs LDA implementations play well with each other?
If the aim is to not have separate LDA classes, on first thought I think maybe then LearningState would be where to put the work in. So, that would mean creating the LearningState trait, then within LDA, there would be multiple definitions of classes that satisfy LearningState. So you might have GibbsLearningState and EMLearningState (solves that naming problem easily). It seems like most of the implementation would be within LearningState anyway, not per se within LDA.
This has the advantage you named, in the LDA constructor we could set the default LearningState type to either EM or Gibbs (probably whichever performs better), allowing users to specify which algorithm they would like if wanted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72609253
LGTM. Merged into master. Thanks everyone for collaborating on LDA! @jkbradley Please create follow-up JIRAs and see who are interested in working on LDA features.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by EntilZha <gi...@git.apache.org>.
Github user EntilZha commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23501104
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * WARNING: If vocabSize and k are large, this can return a large object!
+ *
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(): Array[Array[(Double, String)]] = describeTopics(vocabSize)
+
+ /* TODO
+ * Compute the log likelihood of the observed tokens, given the current parameter estimates:
+ * log P(docs | topics, topic distributions for docs, alpha, eta)
+ *
+ * Note:
+ * - This excludes the prior.
+ * - Even with the prior, this is NOT the same as the data log likelihood given the
+ * hyperparameters.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated log likelihood of the data under this model
+ */
+ // def logLikelihood(documents: RDD[Document]): Double
+
+ /* TODO
+ * Compute the estimated topic distribution for each document.
+ * This is often called “theta” in the literature.
+ *
+ * @param documents A set of documents, where each is represented as a vector of term counts.
+ * This must use the same vocabulary (ordering of term counts) as in training.
+ * Document IDs must be >= 0.
+ * @return Estimated topic distribution for each document.
+ * The returned RDD may be zipped with the given RDD, where each returned vector
+ * is a multinomial distribution over topics.
+ */
+ // def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)]
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Local LDA model.
+ * This model stores only the inferred topics.
+ * It may be used for computing topics for new documents, but it may give less accurate answers
+ * than the [[DistributedLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ *
+ * @param topics Inferred topics (vocabSize x k matrix).
+ */
+@DeveloperApi
+class LocalLDAModel private[clustering] (
+ private val topics: Matrix) extends LDAModel with Serializable {
+
+ import LDA._
+
+ override def k: Int = topics.numCols
+
+ override def vocabSize: Int = topics.numRows
+
+ override def topicsMatrix: Matrix = topics
+
+ override def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]] = {
+ val brzTopics = topics.toBreeze.toDenseMatrix
+ val topicSummary = Range(0, k).map { topicIndex =>
+ val topic = normalize(brzTopics(::, topicIndex), 1.0)
+ topic.toArray.zipWithIndex.sortBy(-_._1).take(maxTermsPerTopic)
+ }.toArray
+ topicSummary.map { topic =>
+ topic.map { case (weight, term) => (weight, term.toString) }
+ }
+ }
+
+ // TODO
+ // override def logLikelihood(documents: RDD[Document]): Double = ???
+
+ // TODO:
+ // override def topicDistributions(documents: RDD[Document]): RDD[(Long, Vector)] = ???
+
+}
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Distributed LDA model.
+ * This model stores the inferred topics, the full training dataset, and the topic distributions.
+ * When computing topics for new documents, it may give more accurate answers
+ * than the [[LocalLDAModel]].
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class DistributedLDAModel private[clustering] (
+ private val state: LDA.LearningState) extends LDAModel {
+
+ import LDA._
+
+ def toLocal: LocalLDAModel = new LocalLDAModel(topicsMatrix)
+
+ override def k: Int = state.k
+
+ override def vocabSize: Int = state.vocabSize
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ *
+ * WARNING: This matrix is collected from an RDD. Beware memory usage when vocabSize, k are large.
+ */
+ override lazy val topicsMatrix: Matrix = {
--- End diff --
This would change to
```scala
state.topicsMatrix
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23729251
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
+ val termSmoothing = if (this.termSmoothing > 0) {
+ this.termSmoothing
+ } else {
+ 1.1
+ }
+ var state = LDA.initialState(documents, k, topicSmoothing, termSmoothing, seed)
+ var iter = 0
+ while (iter < maxIterations) {
+ state = state.next()
+ iter += 1
+ }
+ new DistributedLDAModel(state)
+ }
+}
+
+
+object LDA {
+
+ /*
+ DEVELOPERS NOTE:
+
+ This implementation uses GraphX, where the graph is bipartite with 2 types of vertices:
+ - Document vertices
+ - indexed {0, 1, ..., numDocuments-1}
+ - Store vectors of length k (# topics).
+ - Term vertices
+ - indexed {-1, -2, ..., -vocabSize}
+ - Store vectors of length k (# topics).
+ - Edges correspond to terms appearing in documents.
+ - Edges are directed Document -> Term.
+ - Edges are partitioned by documents.
+
+ Info on EM implementation.
+ - We follow Section 2.2 from Asuncion et al., 2009. We use some of their notation.
+ - In this implementation, there is one edge for every unique term appearing in a document,
+ i.e., for every unique (document, term) pair.
+ - Notation:
+ - N_{wkj} = count of tokens of term w currently assigned to topic k in document j
+ - N_{*} where * is missing a subscript w/k/j is the count summed over missing subscript(s)
+ - gamma_{wjk} = P(z_i = k | x_i = w, d_i = j),
+ the probability of term x_i in document d_i having topic z_i.
+ - Data graph
+ - Document vertices store N_{kj}
+ - Term vertices store N_{wk}
+ - Edges store N_{wj}.
+ - Global data N_k
+ - Algorithm
+ - Initial state:
+ - Document and term vertices store random counts N_{wk}, N_{kj}.
+ - E-step: For each (document,term) pair i, compute P(z_i | x_i, d_i).
+ - Aggregate N_k from term vertices.
+ - Compute gamma_{wjk} for each possible topic k, from each triplet.
+ using inputs N_{wk}, N_{kj}, N_k.
+ - M-step: Compute sufficient statistics for hidden parameters phi and theta
+ (counts N_{wk}, N_{kj}, N_k).
+ - Document update:
+ - N_{kj} <- sum_w N_{wj} gamma_{wjk}
+ - N_j <- sum_k N_{kj} (only needed to output predictions)
+ - Term update:
+ - N_{wk} <- sum_j N_{wj} gamma_{wjk}
+ - N_k <- sum_w N_{wk}
+ */
+
+ /**
+ * :: DeveloperApi ::
+ *
+ * Document with an ID.
+ *
+ * @param counts Vector of term (word) counts in the document.
+ * This is the "bag of words" representation.
+ * @param id Unique ID associated with this document.
+ * Documents should be indexed {0, 1, ..., numDocuments-1}.
+ *
+ * TODO: Can we remove the id and still be able to zip predicted topics with the Documents?
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may
+ * undergo API changes.
+ */
+ @DeveloperApi
+ case class Document(counts: Vector, id: Long)
--- End diff --
Just try to reduce the number of public APIs:) If we take `RDD[Document]`, users need to import it and remember where it is before importing it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by pwendell <gi...@git.apache.org>.
Github user pwendell commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72579348
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-71946376
@EntilZha `LearningState` is not a public API. We can change it freely in the future, so let's not let it block this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23283001
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
--- End diff --
Minor thing. How about moving the if-else to `getTermSmoothing`? For logic separation and also users can have a interface to collect the para
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23322992
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDA.scala ---
@@ -0,0 +1,472 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import breeze.linalg.{DenseVector => BDV, sum => brzSum, normalize, axpy => brzAxpy}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.graphx._
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.Utils
+
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
+ *
+ * Terminology:
+ * - "word" = "term": an element of the vocabulary
+ * - "token": instance of a term appearing in a document
+ * - "topic": multinomial distribution over words representing some concept
+ *
+ * Currently, the underlying implementation uses Expectation-Maximization (EM), implemented
+ * according to the Asuncion et al. (2009) paper referenced below.
+ *
+ * References:
+ * - Original LDA paper (journal version):
+ * Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.
+ * - This class implements their "smoothed" LDA model.
+ * - Paper which clearly explains several algorithms, including EM:
+ * Asuncion, Welling, Smyth, and Teh.
+ * "On Smoothing and Inference for Topic Models." UAI, 2009.
+ *
+ * NOTE: This is currently marked DeveloperApi since it is under active development and may undergo
+ * API changes.
+ */
+@DeveloperApi
+class LDA private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var topicSmoothing: Double,
+ private var termSmoothing: Double,
+ private var seed: Long) extends Logging {
+
+ import LDA._
+
+ def this() = this(k = 10, maxIterations = 20, topicSmoothing = -1, termSmoothing = -1,
+ seed = Utils.random.nextLong())
+
+ /**
+ * Number of topics to infer. I.e., the number of soft cluster centers.
+ * (default = 10)
+ */
+ def getK: Int = k
+
+ def setK(k: Int): this.type = {
+ require(k > 0, s"LDA k (number of clusters) must be > 0, but was set to $k")
+ this.k = k
+ this
+ }
+
+ /**
+ * Topic smoothing parameter (commonly named "alpha").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-document topic distributions
+ * ("theta"). We use a symmetric Dirichlet prior.
+ *
+ * This value should be > 0.0, where larger values mean more smoothing (more regularization).
+ * If set to -1, then topicSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = (50 / k) + 1.
+ * - The 50/k is common in LDA libraries.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTopicSmoothing: Double = topicSmoothing
+
+ def setTopicSmoothing(topicSmoothing: Double): this.type = {
+ require(topicSmoothing > 0.0 || topicSmoothing == -1.0,
+ s"LDA topicSmoothing must be > 0 (or -1 for auto), but was set to $topicSmoothing")
+ if (topicSmoothing > 0.0 && topicSmoothing <= 1.0) {
+ logWarning(s"LDA.topicSmoothing was set to $topicSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.topicSmoothing = topicSmoothing
+ this
+ }
+
+ /**
+ * Term smoothing parameter (commonly named "eta").
+ *
+ * This is the parameter to the Dirichlet prior placed on the per-topic word distributions
+ * (which are called "beta" in the original LDA paper by Blei et al., but are called "phi" in many
+ * later papers such as Asuncion et al., 2009.)
+ *
+ * This value should be > 0.0.
+ * If set to -1, then termSmoothing is set automatically.
+ * (default = -1 = automatic)
+ *
+ * Automatic setting of parameter:
+ * - For EM: default = 0.1 + 1.
+ * - The 0.1 gives a small amount of smoothing.
+ * - The +1 follows Asuncion et al. (2009), who recommend a +1 adjustment for EM.
+ */
+ def getTermSmoothing: Double = termSmoothing
+
+ def setTermSmoothing(termSmoothing: Double): this.type = {
+ require(termSmoothing > 0.0 || termSmoothing == -1.0,
+ s"LDA termSmoothing must be > 0 (or -1 for auto), but was set to $termSmoothing")
+ if (termSmoothing > 0.0 && termSmoothing <= 1.0) {
+ logWarning(s"LDA.termSmoothing was set to $termSmoothing, but for EM, we recommend > 1.0")
+ }
+ this.termSmoothing = termSmoothing
+ this
+ }
+
+ /**
+ * Maximum number of iterations for learning.
+ * (default = 20)
+ */
+ def getMaxIterations: Int = maxIterations
+
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /** Random seed */
+ def getSeed: Long = seed
+
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Learn an LDA model using the given dataset.
+ *
+ * @param documents RDD of documents, where each document is represented as a vector of term
+ * counts plus an ID. Document IDs must be >= 0.
+ * @return Inferred LDA model
+ */
+ def run(documents: RDD[Document]): DistributedLDAModel = {
+ val topicSmoothing = if (this.topicSmoothing > 0) {
+ this.topicSmoothing
+ } else {
+ (50.0 / k) + 1.0
+ }
--- End diff --
Good point, will do.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-70571871
@EntilZha I agree with your sketch: abstracting LearningState, and having different versions for each algorithm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/4047#discussion_r23727987
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/LDAModel.scala ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseMatrix => BDM, normalize}
+
+import org.apache.spark.annotation.DeveloperApi
+import org.apache.spark.mllib.linalg.{Vectors, Vector, Matrices, Matrix}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.BoundedPriorityQueue
+
+/**
+ * :: DeveloperApi ::
+ *
+ * Latent Dirichlet Allocation (LDA) model.
+ *
+ * This abstraction permits for different underlying representations,
+ * including local and distributed data structures.
+ */
+@DeveloperApi
+abstract class LDAModel private[clustering] {
+
+ import LDA._
+
+ /** Number of topics */
+ def k: Int
+
+ /** Vocabulary size (number of terms or terms in the vocabulary) */
+ def vocabSize: Int
+
+ /**
+ * Inferred topics, where each topic is represented by a distribution over terms.
+ * This is a matrix of size vocabSize x k, where each column is a topic.
+ * No guarantees are given about the ordering of the topics.
+ */
+ def topicsMatrix: Matrix
+
+ /**
+ * Return the topics described by weighted terms.
+ *
+ * This limits the number of terms per topic.
+ * This is approximate; it may not return exactly the top-weighted terms for each topic.
+ * To get a more precise set of top terms, increase maxTermsPerTopic.
+ *
+ * @param maxTermsPerTopic Maximum number of terms to collect for each topic.
+ * @return Array over topics, where each element is a set of top terms represented
+ * as (term weight in topic, term index).
+ * Each topic's terms are sorted in order of decreasing weight.
+ */
+ def describeTopics(maxTermsPerTopic: Int): Array[Array[(Double, String)]]
--- End diff --
I'm OK with different APIs, but I really hope we can figure out a good way to make it easy for users to use either pre-indexed terms or plain text.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/4047#issuecomment-72265733
[Test build #26421 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26421/consoleFull) for PR 4047 at commit [`1db89e2`](https://github.com/apache/spark/commit/1db89e2964aee34f9c33300be271dde41d61a782).
* This patch merges cleanly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org