You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by hhbyyh <gi...@git.apache.org> on 2015/09/01 11:39:12 UTC
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
GitHub user hhbyyh opened a pull request:
https://github.com/apache/spark/pull/8551
[SPARK-10393] use ML pipeline in LDA example
jira: https://issues.apache.org/jira/browse/SPARK-10393
Since the logic of the text processing part has been moved to ML estimators/transformers, replace the related code in LDA Example with the ML pipeline.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/hhbyyh/spark ldaExUpdate
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/8551.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #8551
----
commit 84cc1beb5852ceab635fbf95f64b17a1c7b4046e
Author: Yuhao Yang <hh...@gmail.com>
Date: 2015-09-01T09:32:28Z
update lda example
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147305874
[Test build #43561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/consoleFull) for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162653524
**[Test build #2177 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2177/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147304609
@feynmanliang @jkbradley Sorry for the late update.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155264646
unrelated UT timeout failure:
SPARK-11188 Analysis error reporting *** FAILED *** (2 minutes)
[info] =======================
[info] CliSuite failure output
[info] =======================
[info] Spark SQL CLI command line: ../../bin/spark-sql --master local --hiveconf
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r39460880
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -186,121 +186,52 @@ object LDAExample {
* Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
* @return (corpus, vocabulary as array, total token count in corpus)
*/
- private def preprocess(
+ private def preProcess(
sc: SparkContext,
paths: Seq[String],
vocabSize: Int,
- stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ stopWordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val sqlContext = new SQLContext(sc)
+ import sqlContext.implicits._
+
+ val df = textRDD.toDF("texts")
--- End diff --
After moving where implicit import is, this can just be `sc.textFile(paths.mkString(",")).toDF("texts")`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162822103
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47324/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162822102
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155378185
**[Test build #45515 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45515/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155376709
Jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r46874402
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -18,9 +18,9 @@
// scalastyle:off println
package org.apache.spark.examples.mllib
-import java.text.BreakIterator
-
-import scala.collection.mutable
+import org.apache.spark.ml.Pipeline
--- End diff --
Organize imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136654355
[Test build #41868 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41868/consoleFull) for PR 8551 at commit [`84cc1be`](https://github.com/apache/spark/commit/84cc1beb5852ceab635fbf95f64b17a1c7b4046e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r39460809
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -186,121 +186,52 @@ object LDAExample {
* Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
* @return (corpus, vocabulary as array, total token count in corpus)
*/
- private def preprocess(
+ private def preProcess(
sc: SparkContext,
paths: Seq[String],
vocabSize: Int,
- stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ stopWordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val sqlContext = new SQLContext(sc)
--- End diff --
`SQLContext.getOrCreate`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136662532
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155240352
**[Test build #2026 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2026/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r39460759
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -186,121 +186,52 @@ object LDAExample {
* Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
* @return (corpus, vocabulary as array, total token count in corpus)
*/
- private def preprocess(
+ private def preProcess(
--- End diff --
Can we revert these NO_OP changes (here and `stopwordFile`)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-140443465
@feynmanliang Thanks for the review. I'll update later today.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155384289
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155384291
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/45515/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162660020
**[Test build #2177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2177/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r39460900
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -186,121 +186,52 @@ object LDAExample {
* Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
* @return (corpus, vocabulary as array, total token count in corpus)
*/
- private def preprocess(
+ private def preProcess(
sc: SparkContext,
paths: Seq[String],
vocabSize: Int,
- stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ stopWordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val sqlContext = new SQLContext(sc)
+ import sqlContext.implicits._
+
+ val df = textRDD.toDF("texts")
--- End diff --
Also a small nit: I would call them `docs` instead of `texts` to be consistent with user guide
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155253327
**[Test build #2026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/2026/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/8551
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155377078
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162822008
**[Test build #47324 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47324/consoleFull)** for PR 8551 at commit [`672ce97`](https://github.com/apache/spark/commit/672ce97892c974994630527e0e724090b3ac52e5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147310348
[Test build #43561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/console) for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147310378
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43561/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147304561
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r46874564
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -192,115 +192,45 @@ object LDAExample {
vocabSize: Int,
stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ val sqlContext = SQLContext.getOrCreate(sc)
+ import sqlContext.implicits._
+
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
- val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val df = sc.textFile(paths.mkString(",")).toDF("docs")
+ val customizedStopWords: Array[String] = if (stopwordFile.isEmpty) {
+ Array.empty[String]
+ } else {
+ val stopWordText = sc.textFile(stopwordFile).collect()
+ stopWordText.flatMap(_.stripMargin.split("\\s+"))
}
-
- val documents = tokenized.map { case (id, tokens) =>
- // Filter tokens by vocabulary, and create word count vector representation of document.
- val wc = new mutable.HashMap[Int, Int]()
- tokens.foreach { term =>
- if (vocab.contains(term)) {
- val termIndex = vocab(term)
- wc(termIndex) = wc.getOrElse(termIndex, 0) + 1
- }
- }
- val indices = wc.keys.toArray.sorted
- val values = indices.map(i => wc(i).toDouble)
-
- val sb = Vectors.sparse(vocab.size, indices, values)
- (id, sb)
- }
-
- val vocabArray = new Array[String](vocab.size)
- vocab.foreach { case (term, i) => vocabArray(i) = term }
-
- (documents, vocabArray, selectedTokenCount)
+ val tokenizer = new RegexTokenizer()
+ .setInputCol("docs")
+ .setOutputCol("rawTokens")
+ val stopWordsRemover = new StopWordsRemover()
+ .setInputCol("rawTokens")
+ .setOutputCol("tokens")
+ stopWordsRemover.setStopWords(stopWordsRemover.getStopWords ++ customizedStopWords)
+ val countVectorizer = new CountVectorizer()
+ .setVocabSize(vocabSize)
+ .setInputCol("tokens")
+ .setOutputCol("vectors")
--- End diff --
How about calling these "features" instead of "vectors"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136662449
[Test build #41868 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41868/console) for PR 8551 at commit [`84cc1be`](https://github.com/apache/spark/commit/84cc1beb5852ceab635fbf95f64b17a1c7b4046e).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class DCT(JavaTransformer, HasInputCol, HasOutputCol):`
* `class SQLTransformer(JavaTransformer):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147310376
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147488013
need to trigger the test again?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162665829
Those 3 items (thanks @BenFradet ) are the only issues I see. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155384164
**[Test build #45515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45515/consoleFull)** for PR 8551 at commit [`ee4832d`](https://github.com/apache/spark/commit/ee4832d512fdeb57773f51846c9e75ecd8774c01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by hhbyyh <gi...@git.apache.org>.
Github user hhbyyh commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162812998
Thanks @jkbradley @BenFradet for the review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136662537
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41868/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162815826
**[Test build #47324 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47324/consoleFull)** for PR 8551 at commit [`672ce97`](https://github.com/apache/spark/commit/672ce97892c974994630527e0e724090b3ac52e5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-147304573
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-155377106
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136651324
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-136651368
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by BenFradet <gi...@git.apache.org>.
Github user BenFradet commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r46874064
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -192,115 +192,45 @@ object LDAExample {
vocabSize: Int,
stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ val sqlContext = SQLContext.getOrCreate(sc)
+ import sqlContext.implicits._
+
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
- val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val df = sc.textFile(paths.mkString(",")).toDF("docs")
+ val customizedStopWords: Array[String] = if (stopwordFile.isEmpty) {
+ Array.empty[String]
+ } else {
+ val stopWordText = sc.textFile(stopwordFile).collect()
+ stopWordText.flatMap(_.stripMargin.split("\\s+"))
}
-
- val documents = tokenized.map { case (id, tokens) =>
- // Filter tokens by vocabulary, and create word count vector representation of document.
- val wc = new mutable.HashMap[Int, Int]()
- tokens.foreach { term =>
- if (vocab.contains(term)) {
- val termIndex = vocab(term)
- wc(termIndex) = wc.getOrElse(termIndex, 0) + 1
- }
- }
- val indices = wc.keys.toArray.sorted
- val values = indices.map(i => wc(i).toDouble)
-
- val sb = Vectors.sparse(vocab.size, indices, values)
- (id, sb)
- }
-
- val vocabArray = new Array[String](vocab.size)
- vocab.foreach { case (term, i) => vocabArray(i) = term }
-
- (documents, vocabArray, selectedTokenCount)
+ val tokenizer = new RegexTokenizer()
+ .setInputCol("docs")
+ .setOutputCol("rawTokens")
+ val stopWordsRemover = new StopWordsRemover()
+ .setInputCol("rawTokens")
+ .setOutputCol("tokens")
+ stopWordsRemover.setStopWords(stopWordsRemover.getStopWords ++ customizedStopWords)
+ val countVectorizer = new CountVectorizer()
+ .setVocabSize(vocabSize)
+ .setInputCol("tokens")
+ .setOutputCol("vectors")
+
+ val pipeline = new Pipeline()
+ .setStages(Array(tokenizer, stopWordsRemover, countVectorizer))
+
+ val model = pipeline.fit(df)
+ val documents = model.transform(df)
+ .select("vectors")
+ .map { case Row(features: Vector) => features }
+ .zipWithIndex()
+ .map(_.swap)
+
+ (documents,
+ model.stages(2).asInstanceOf[CountVectorizerModel].vocabulary, // vocabulary
+ documents.map(_._2.numActives).sum().toLong) // total token count
}
}
-/**
- * Simple Tokenizer.
- *
- * TODO: Formalize the interface, and make this a public class in mllib.feature
- */
-private class SimpleTokenizer(sc: SparkContext, stopwordFile: String) extends Serializable {
-
- private val stopwords: Set[String] = if (stopwordFile.isEmpty) {
- Set.empty[String]
- } else {
- val stopwordText = sc.textFile(stopwordFile).collect()
- stopwordText.flatMap(_.stripMargin.split("\\s+")).toSet
- }
-
- // Matches sequences of Unicode letters
- private val allWordRegex = "^(\\p{L}*)$".r
-
- // Ignore words shorter than this length.
- private val minWordLength = 3
-
- def getWords(text: String): IndexedSeq[String] = {
-
- val words = new mutable.ArrayBuffer[String]()
-
- // Use Java BreakIterator to tokenize text into words.
- val wb = BreakIterator.getWordInstance
- wb.setText(text)
-
- // current,end index start,end of each word
- var current = wb.first()
- var end = wb.next()
- while (end != BreakIterator.DONE) {
- // Convert to lowercase
- val word: String = text.substring(current, end).toLowerCase
- // Remove short words and strings that aren't only letters
- word match {
- case allWordRegex(w) if w.length >= minWordLength && !stopwords.contains(w) =>
- words += w
- case _ =>
- }
-
- current = end
- try {
- end = wb.next()
- } catch {
- case e: Exception =>
- // Ignore remaining text in line.
- // This is a known bug in BreakIterator (for some Java versions),
- // which fails when it sees certain characters.
- end = BreakIterator.DONE
- }
- }
- words
- }
-
-}
-// scalastyle:on println
--- End diff --
Are you sure about removing the scalastyle line?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/8551#issuecomment-162970671
LGTM. Merging with master and branch-1.6
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-10393] use ML pipeline in LDA example
Posted by feynmanliang <gi...@git.apache.org>.
Github user feynmanliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/8551#discussion_r39460837
--- Diff: examples/src/main/scala/org/apache/spark/examples/mllib/LDAExample.scala ---
@@ -186,121 +186,52 @@ object LDAExample {
* Load documents, tokenize them, create vocabulary, and prepare documents as term count vectors.
* @return (corpus, vocabulary as array, total token count in corpus)
*/
- private def preprocess(
+ private def preProcess(
sc: SparkContext,
paths: Seq[String],
vocabSize: Int,
- stopwordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
+ stopWordFile: String): (RDD[(Long, Vector)], Array[String], Long) = {
// Get dataset of document texts
// One document per line in each text file. If the input consists of many small files,
// this can result in a large number of small partitions, which can degrade performance.
// In this case, consider using coalesce() to create fewer, larger partitions.
val textRDD: RDD[String] = sc.textFile(paths.mkString(","))
-
- // Split text into words
- val tokenizer = new SimpleTokenizer(sc, stopwordFile)
- val tokenized: RDD[(Long, IndexedSeq[String])] = textRDD.zipWithIndex().map { case (text, id) =>
- id -> tokenizer.getWords(text)
- }
- tokenized.cache()
-
- // Counts words: RDD[(word, wordCount)]
- val wordCounts: RDD[(String, Long)] = tokenized
- .flatMap { case (_, tokens) => tokens.map(_ -> 1L) }
- .reduceByKey(_ + _)
- wordCounts.cache()
- val fullVocabSize = wordCounts.count()
- // Select vocab
- // (vocab: Map[word -> id], total tokens after selecting vocab)
- val (vocab: Map[String, Int], selectedTokenCount: Long) = {
- val tmpSortedWC: Array[(String, Long)] = if (vocabSize == -1 || fullVocabSize <= vocabSize) {
- // Use all terms
- wordCounts.collect().sortBy(-_._2)
- } else {
- // Sort terms to select vocab
- wordCounts.sortBy(_._2, ascending = false).take(vocabSize)
- }
- (tmpSortedWC.map(_._1).zipWithIndex.toMap, tmpSortedWC.map(_._2).sum)
+ val sqlContext = new SQLContext(sc)
+ import sqlContext.implicits._
--- End diff --
nit: I would put this sqlContext instantiation and implicits import right at the start of the method (on L194)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org