You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zhengruifeng <gi...@git.apache.org> on 2016/03/26 04:52:24 UTC

[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

GitHub user zhengruifeng opened a pull request:

    https://github.com/apache/spark/pull/11974

    [SPARK-14174][MLLIB] Accelerate KMeans via Mini-Batch EM

    ## What changes were proposed in this pull request?
    add the support of MiniBatch EM into KMeans
    
    
    ## How was this patch tested?
    manual tests

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/zhengruifeng/spark mbkm

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/11974.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #11974
    
----
commit 2511f5e1ffe13f0422d42ca2f9c1cb2e7dab1410
Author: Ruifeng Zheng <ru...@foxmail.com>
Date:   2016-03-25T06:46:04Z

    add minibatch-kmeans

commit b1dac5493d318a4e4a9ba9f3ac2a08aad50e4db0
Author: Ruifeng Zheng <ru...@foxmail.com>
Date:   2016-03-25T07:00:03Z

    fix

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77244 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77244/testReport)** for PR 11974 at commit [`bf838ab`](https://github.com/apache/spark/commit/bf838ab6c668986fe9bc1adadb29bef2214edebf).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batc...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r118548583
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala ---
    @@ -85,6 +85,20 @@ private[clustering] trait KMeansParams extends Params with HasMaxIter with HasFe
       def getInitSteps: Int = $(initSteps)
     
       /**
    +   * The fraction of the data to update centers per iteration. Must be &gt; 0 and &le; 1.
    --- End diff --
    
    nit: "fraction of data used to ..."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-203947479
  
    **[Test build #54625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54625/consoleFull)** for PR 11974 at commit [`10eb878`](https://github.com/apache/spark/commit/10eb8788f46d932fa9c374a6da6b46d17de11d5a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batc...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r118546602
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/clustering/KMeansSuite.scala ---
    @@ -89,6 +92,9 @@ class KMeansSuite extends SparkFunSuite with MLlibTestSparkContext with DefaultR
         intercept[IllegalArgumentException] {
           new KMeans().setInitSteps(0)
         }
    +    intercept[IllegalArgumentException] {
    +      new KMeans().setMiniBatchFraction(0)
    --- End diff --
    
    Minor nit but we should probably check a few edge cases - <0, >1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-203962433
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #70743 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70743/testReport)** for PR 11974 at commit [`d6472ea`](https://github.com/apache/spark/commit/d6472ea0a4e6df259ec1e143e2cb98079b243a3c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-203962086
  
    **[Test build #54625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54625/consoleFull)** for PR 11974 at commit [`10eb878`](https://github.com/apache/spark/commit/10eb8788f46d932fa9c374a6da6b46d17de11d5a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-205748375
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54978/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #69402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69402/consoleFull)** for PR 11974 at commit [`c27f128`](https://github.com/apache/spark/commit/c27f128136817fed810602df4ccf7ed6e77894c9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77287/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Jenkins, retest this please


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    I commented on the JIRA about posting performance test results. 
    
    This should also be updated for `@Since` tags to `2.3.0`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #70743 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/70743/testReport)** for PR 11974 at commit [`d6472ea`](https://github.com/apache/spark/commit/d6472ea0a4e6df259ec1e143e2cb98079b243a3c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-203962436
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54625/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-205734679
  
    **[Test build #54978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54978/consoleFull)** for PR 11974 at commit [`10eb878`](https://github.com/apache/spark/commit/10eb8788f46d932fa9c374a6da6b46d17de11d5a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batc...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r118548684
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala ---
    @@ -85,6 +85,20 @@ private[clustering] trait KMeansParams extends Params with HasMaxIter with HasFe
       def getInitSteps: Int = $(initSteps)
     
       /**
    +   * The fraction of the data to update centers per iteration. Must be &gt; 0 and &le; 1.
    +   * Default: 1.0.
    +   * @group param
    +   */
    +  @Since("2.3.0")
    +  final val miniBatchFraction = new DoubleParam(this, "miniBatchFraction", "The fraction of the" +
    +    " data to update clustering centers per iteration. Must be in (0, 1].",
    --- End diff --
    
    ditto here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77397/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML][WIP] Accelerate KMeans via Mini-Batch ...

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @MLnick  @sethah    According to the [paper](https://www.eecs.tufts.edu/%7Edsculley/papers/fastkmeans.pdf), `MiniBatchKMeans` with `fraction=1.0` is not equal to `KMeans`, and they are quite different two algorithms that we should implement different logic for them.
    So I think we should make `MiniBatchKMeans` a separate estimator, and can close this pr for now.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/70743/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201730249
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @MLnick `@Since` updated. And the performance test result is attached in the JIRA. Thanks for your reviewing!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77244 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77244/testReport)** for PR 11974 at commit [`bf838ab`](https://github.com/apache/spark/commit/bf838ab6c668986fe9bc1adadb29bef2214edebf).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/77244/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @sethah Agree that if I/O is the bottleneck, the speedup should be small.
    The cost in results doc is computed on the whole dataset, no the sampled ones. Since I think comparing cost on different instances is not fair. 
    I did not run MiniBatchKmeans with a large iterations, just because I didn't want to make too many copy/paste operations to fill the tables...
    Following is another two tests with `k=10` , `maxIterations=20` and `frac=1.0, 0.1, 0.01, 0.001` (on my laptop, tests in jira is performed on a server)
    
    Low dense data
    ```
    scala> val n = 1000000
    n: Int = 1000000
    
    scala> val dim = 10
    dim: Int = 10
    
    scala> val rdd = sc.parallelize(1 to n).map(i => Vectors.dense(Array.fill(dim)(random.nextDouble()))).persist()
    rdd: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[1] at map at <console>:41
    
    scala> rdd.count
    res0: Long = 1000000         
    
    scala> val rdd = sc.parallelize(1 to n).map(i => Vectors.dense(Array.fill(dim)(random.nextDouble()))).persist()
    rdd: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[1] at map at <console>:41
    
    scala> rdd.count
    res0: Long = 1000000                                                            
    
    scala> Seq(1.0, 0.1, 0.01, 0.001).foreach{f => println(MiniBatchKMeans.train(rdd, k=10, maxIterations=20, "random", f, 123).computeCost(rdd))}
    17/05/26 09:14:09 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
    17/05/26 09:14:09 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
    Iter 0: Duration 2004, cost 788582.9738726234                                   
    Iter 1: Duration 833, cost 630535.2796758788                                    
    Iter 2: Duration 540, cost 614845.1513723353
    Iter 3: Duration 605, cost 610066.9062292713
    Iter 4: Duration 561, cost 608031.849076637
    Iter 5: Duration 556, cost 606823.855125578
    Iter 6: Duration 573, cost 606045.5575892355
    Iter 7: Duration 476, cost 605559.3562866914
    Iter 8: Duration 477, cost 605256.6012606357
    Iter 9: Duration 485, cost 605057.8933848375
    Iter 10: Duration 444, cost 604913.8680054378
    Iter 11: Duration 496, cost 604799.3876557952
    Iter 12: Duration 380, cost 604700.0366582843
    Iter 13: Duration 484, cost 604611.0252890466
    Iter 14: Duration 412, cost 604523.1312451813
    Iter 15: Duration 469, cost 604437.1852339945
    Iter 16: Duration 414, cost 604350.2820076608
    Iter 17: Duration 425, cost 604269.8453690736
    Iter 18: Duration 422, cost 604191.0100384927
    Iter 19: Duration 552, cost 604115.8521715086
    604045.2371941482
    Iter 0: Duration 107, cost 78968.67203555592
    Iter 1: Duration 161, cost 62982.950117238244
    Iter 2: Duration 145, cost 61679.984194278164
    Iter 3: Duration 182, cost 61180.225915234245
    Iter 4: Duration 153, cost 60774.909568730814
    Iter 5: Duration 140, cost 60499.18450499466
    Iter 6: Duration 111, cost 60631.84303512169
    Iter 7: Duration 140, cost 60293.24798484023
    Iter 8: Duration 114, cost 60586.99701309156
    Iter 9: Duration 151, cost 60748.94599207808
    Iter 10: Duration 148, cost 60325.21264819525
    Iter 11: Duration 169, cost 60661.49913084767
    Iter 12: Duration 149, cost 60589.54155054191
    Iter 13: Duration 103, cost 60388.33938789272
    Iter 14: Duration 115, cost 60682.58364074669
    Iter 15: Duration 108, cost 60477.18379844011
    Iter 16: Duration 125, cost 60692.99362376495
    Iter 17: Duration 125, cost 60457.083009454785
    Iter 18: Duration 151, cost 60155.58402551571
    Iter 19: Duration 149, cost 60675.606410578795
    604111.2208211605
    Iter 0: Duration 76, cost 7806.9827458185455
    Iter 1: Duration 98, cost 6324.535452971191
    Iter 2: Duration 104, cost 6202.206415357503
    Iter 3: Duration 110, cost 6072.991331049303
    Iter 4: Duration 102, cost 6152.855401385752
    Iter 5: Duration 153, cost 6141.858875200935
    Iter 6: Duration 129, cost 5987.641548753511
    Iter 7: Duration 100, cost 6074.370232167772
    Iter 8: Duration 93, cost 6087.813342965538
    Iter 9: Duration 95, cost 6048.723797127357
    Iter 10: Duration 81, cost 6068.360272434616
    Iter 11: Duration 77, cost 6026.502400302613
    Iter 12: Duration 113, cost 6075.433877179167
    Iter 13: Duration 110, cost 6116.414343219921
    Iter 14: Duration 94, cost 6115.643402410937
    Iter 15: Duration 120, cost 6018.897745377334
    Iter 16: Duration 107, cost 6073.868377665494
    Iter 17: Duration 121, cost 6049.590697175236
    Iter 18: Duration 101, cost 6002.554790134795
    Iter 19: Duration 107, cost 6182.314365817103
    604478.1009599782
    Iter 0: Duration 84, cost 720.0194354140257
    Iter 1: Duration 96, cost 628.575082152244
    Iter 2: Duration 93, cost 647.4085507383511
    Iter 3: Duration 94, cost 586.6110099943827
    Iter 4: Duration 73, cost 606.2971088014147
    Iter 5: Duration 77, cost 561.5283061240732
    Iter 6: Duration 92, cost 591.4893772291216
    Iter 7: Duration 100, cost 589.4491867389609
    Iter 8: Duration 94, cost 606.4013710450262
    Iter 9: Duration 98, cost 599.275037352022
    Iter 10: Duration 100, cost 633.685773790703
    Iter 11: Duration 90, cost 600.3590843929948
    Iter 12: Duration 100, cost 587.7426871534223
    Iter 13: Duration 110, cost 601.867530070673
    Iter 14: Duration 81, cost 582.4792460355222
    Iter 15: Duration 70, cost 566.808139887458
    Iter 16: Duration 103, cost 593.1931284344439
    Iter 17: Duration 75, cost 593.0484823003007
    Iter 18: Duration 87, cost 590.4515560012428
    Iter 19: Duration 110, cost 636.6432124538064
    606774.8878757863
    ```
    
    Low sparse data:
    ```
    scala> val dim = 1000
    dim: Int = 1000
    
    scala> val nnz = 10
    nnz: Int = 10
    
    scala> val rdd = sc.parallelize(1 to n).map(i => Vectors.sparse(dim, random.shuffle(0 to dim - 1).slice(0, nnz).sorted.toArray, Array.fill(nnz)(random.nextDouble()))).persist()
    rdd: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector] = MapPartitionsRDD[243] at map at <console>:43
    
    scala> rdd.count
    res2: Long = 1000000                                                            
    
    scala> 
    
    scala> Seq(1.0, 0.1, 0.01, 0.001).foreach{f => println(MiniBatchKMeans.train(rdd, k=10, maxIterations=20, "random", f, 123).computeCost(rdd))}
    Iter 0: Duration 855, cost 5136755.005959613                                    
    Iter 1: Duration 463, cost 3290593.234145827
    Iter 2: Duration 361, cost 3285988.921198346
    Iter 3: Duration 414, cost 3284159.7181967623
    Iter 4: Duration 412, cost 3283753.486046558
    Iter 5: Duration 359, cost 3283736.9857063256
    Iter 6: Duration 467, cost 3283734.350937253
    Iter 7: Duration 395, cost 3283733.8835221143
    Iter 8: Duration 485, cost 3283733.764217921
    Iter 9: Duration 388, cost 3283733.718858701
    Iter 10: Duration 388, cost 3283733.7100546346
    Iter 11: Duration 472, cost 3283733.706424741
    Iter 12: Duration 343, cost 3283733.7043986535
    Iter 13: Duration 454, cost 3283733.7030791626
    Iter 14: Duration 411, cost 3283733.7022940316
    3283733.7022940316
    Iter 0: Duration 186, cost 515407.20129455277
    Iter 1: Duration 115, cost 328799.0024563563
    Iter 2: Duration 109, cost 329612.0112105651
    Iter 3: Duration 117, cost 329472.7363659147
    Iter 4: Duration 110, cost 328308.1740654686
    Iter 5: Duration 128, cost 327555.0913579955
    Iter 6: Duration 133, cost 328570.3274713976
    Iter 7: Duration 131, cost 326772.2661423224
    Iter 8: Duration 125, cost 329411.9244852358
    Iter 9: Duration 139, cost 329443.76867151144
    Iter 10: Duration 124, cost 328177.790480869
    Iter 11: Duration 129, cost 328689.4495351914
    Iter 12: Duration 117, cost 329209.31353470264
    Iter 13: Duration 178, cost 328004.5349875497
    Iter 14: Duration 176, cost 329325.6991534679
    Iter 15: Duration 150, cost 328495.6369945316
    Iter 16: Duration 105, cost 330078.4435948311
    Iter 17: Duration 142, cost 327892.86837545113
    Iter 18: Duration 102, cost 327830.7592207911
    Iter 19: Duration 101, cost 329399.3734780577
    3284006.927754979
    Iter 0: Duration 74, cost 50827.240443426854
    Iter 1: Duration 89, cost 33098.14517086026
    Iter 2: Duration 77, cost 33310.16139043249
    Iter 3: Duration 88, cost 32502.802014248817
    Iter 4: Duration 98, cost 33314.737208046135
    Iter 5: Duration 102, cost 33413.513205555886
    Iter 6: Duration 101, cost 32513.72389025821
    Iter 7: Duration 112, cost 32844.3953815156
    Iter 8: Duration 105, cost 32983.07571425794
    Iter 9: Duration 101, cost 33060.23970665815
    Iter 10: Duration 102, cost 33210.19304032954
    Iter 11: Duration 74, cost 32761.495919892204
    Iter 12: Duration 81, cost 32968.94719485651
    Iter 13: Duration 101, cost 33236.530002761705
    Iter 14: Duration 97, cost 33148.95273308664
    Iter 15: Duration 101, cost 32854.05889019754
    Iter 16: Duration 118, cost 33125.816622864055
    Iter 17: Duration 101, cost 32967.017394099494
    Iter 18: Duration 90, cost 32625.42645823063
    Iter 19: Duration 118, cost 33449.67241159137
    3291810.8288164968
    Iter 0: Duration 81, cost 4746.085400496897
    Iter 1: Duration 161, cost 3244.616681775365
    Iter 2: Duration 123, cost 3322.3529742161836
    Iter 3: Duration 121, cost 3158.7377760566465
    Iter 4: Duration 113, cost 3269.522708407527
    Iter 5: Duration 85, cost 3102.8306222841898
    Iter 6: Duration 93, cost 3218.01263895013
    Iter 7: Duration 102, cost 3170.8045673871875
    Iter 8: Duration 88, cost 3305.5274012487007
    Iter 9: Duration 82, cost 3268.0629858657344
    Iter 10: Duration 71, cost 3491.1456725640255
    Iter 11: Duration 78, cost 3297.524532173714
    Iter 12: Duration 84, cost 3289.097873512691
    Iter 13: Duration 103, cost 3318.1276798335107
    Iter 14: Duration 118, cost 3225.800723626809
    Iter 15: Duration 108, cost 3140.657430809164
    Iter 16: Duration 113, cost 3221.9599112200685
    Iter 17: Duration 109, cost 3305.624135805366
    Iter 18: Duration 86, cost 3195.7532498689884
    Iter 19: Duration 85, cost 3451.1008608814627
    3312203.2361961133
    ```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77287 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77287/testReport)** for PR 11974 at commit [`46d9c7b`](https://github.com/apache/spark/commit/46d9c7bbbfdd663a19212c2ddc7431ccd6293022).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #69402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69402/consoleFull)** for PR 11974 at commit [`c27f128`](https://github.com/apache/spark/commit/c27f128136817fed810602df4ccf7ed6e77894c9).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r57512700
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala ---
    @@ -503,6 +525,39 @@ object KMeans {
        *             function will be returned. (default: 1)
        * @param initializationMode The initialization algorithm. This can either be "random" or
        *                           "k-means||". (default: "k-means||")
    +   * @param miniBatchFraction fraction of the input data set that should be used for
    +   *                          one iteration of EM. Default value 1.0.
    +   * @param seed Random seed for cluster initialization. Default is to generate seed based
    +   *             on system time.
    +   */
    +  @Since("2.0.0")
    +  def train(
    --- End diff --
    
    I don't think we want yet another overload for this parameter. It can be set via the instance methods if desired.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11974: [SPARK-14174][ML][WIP] Accelerate KMeans via Mini...

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng closed the pull request at:

    https://github.com/apache/spark/pull/11974


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML][WIP] Accelerate KMeans via Mini-Batch ...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @sethah the idea is to get to a "good enough" solution in a fraction of the time. The [paper](http://www.eecs.tufts.edu/~dsculley/papers/fastkmeans.pdf) shows it achieving "near optimal" cost in far fewer iterations. Scikit-learn has [MiniBatchKMeans](http://scikit-learn.org/stable/modules/generated/sklearn.cluster.MiniBatchKMeans.html) and shows a comparison with KMeans that indicates the resulting clustering is typically [very similar](http://scikit-learn.org/stable/auto_examples/cluster/plot_mini_batch_kmeans.html).
    
    Though @zhengruifeng the above comparison should probably also compare the "fully converged" full data solution (cost and # iters) with the minibatch solution (cost and # iters). It is not clear from what you posted what that result is?
    
    Actually I also forgot that we will definitely need to update the examples and also add a section in the guide explaining the use of `miniBatchFraction` and explaining the tradeoffs and providing references...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batc...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r118548792
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala ---
    @@ -85,6 +85,20 @@ private[clustering] trait KMeansParams extends Params with HasMaxIter with HasFe
       def getInitSteps: Int = $(initSteps)
     
       /**
    +   * The fraction of the data to update centers per iteration. Must be &gt; 0 and &le; 1.
    +   * Default: 1.0.
    +   * @group param
    +   */
    +  @Since("2.3.0")
    +  final val miniBatchFraction = new DoubleParam(this, "miniBatchFraction", "The fraction of the" +
    +    " data to update clustering centers per iteration. Must be in (0, 1].",
    --- End diff --
    
    and "cluster centers" rather than "clustering" centers


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r57827407
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala ---
    @@ -503,6 +525,39 @@ object KMeans {
        *             function will be returned. (default: 1)
        * @param initializationMode The initialization algorithm. This can either be "random" or
        *                           "k-means||". (default: "k-means||")
    +   * @param miniBatchFraction fraction of the input data set that should be used for
    +   *                          one iteration of EM. Default value 1.0.
    +   * @param seed Random seed for cluster initialization. Default is to generate seed based
    +   *             on system time.
    +   */
    +  @Since("2.0.0")
    +  def train(
    --- End diff --
    
    What about createing another object named MiniBatchKMenas and put the API into it? It is convenient to use it as a static method.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201730227
  
    **[Test build #54258 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54258/consoleFull)** for PR 11974 at commit [`40b7d04`](https://github.com/apache/spark/commit/40b7d04b2d1830623b8536c93b79c06c3250eab6).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77397 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77397/testReport)** for PR 11974 at commit [`a5ce8ca`](https://github.com/apache/spark/commit/a5ce8caad5a81c2e5c6789402748a0b4f15b2fbc).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69402/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201701274
  
    **[Test build #54248 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54248/consoleFull)** for PR 11974 at commit [`b1dac54`](https://github.com/apache/spark/commit/b1dac5493d318a4e4a9ba9f3ac2a08aad50e4db0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201701496
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54248/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-205748368
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-205747943
  
    **[Test build #54978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54978/consoleFull)** for PR 11974 at commit [`10eb878`](https://github.com/apache/spark/commit/10eb8788f46d932fa9c374a6da6b46d17de11d5a).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on a diff in the pull request:

    https://github.com/apache/spark/pull/11974#discussion_r58053990
  
    --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeans.scala ---
    @@ -503,6 +525,39 @@ object KMeans {
        *             function will be returned. (default: 1)
        * @param initializationMode The initialization algorithm. This can either be "random" or
        *                           "k-means||". (default: "k-means||")
    +   * @param miniBatchFraction fraction of the input data set that should be used for
    +   *                          one iteration of EM. Default value 1.0.
    +   * @param seed Random seed for cluster initialization. Default is to generate seed based
    +   *             on system time.
    +   */
    +  @Since("2.0.0")
    +  def train(
    --- End diff --
    
    I also think this API should be remove now. I will remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    cc @srowen @setha also


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #69405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69405/consoleFull)** for PR 11974 at commit [`83b5b8e`](https://github.com/apache/spark/commit/83b5b8e26b21a23a423a991564e28be24ed1551c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201723275
  
    **[Test build #54258 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54258/consoleFull)** for PR 11974 at commit [`40b7d04`](https://github.com/apache/spark/commit/40b7d04b2d1830623b8536c93b79c06c3250eab6).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML][WIP] Accelerate KMeans via Mini-Batch ...

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @MLnick  I mean that the algorithm in the paper is not implemented in current PR. I think I need to update this PR according to the paper, and then retest it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77397 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77397/testReport)** for PR 11974 at commit [`a5ce8ca`](https://github.com/apache/spark/commit/a5ce8caad5a81c2e5c6789402748a0b4f15b2fbc).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201730251
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54258/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Mini-batching in Spark generally isn't that efficient, since to extract a mini-batch you still need to iterate over the entire dataset - and that means reading it from disk if it doesn't fit into memory.
    
    The performance tests posted on the jira are hard to interpret. It looks to me like the computation time goes down as you sample less data, but the cost function doesn't decrease as much. What's the conclusion? I'd be more interested to see how long it takes to get to the same cost, all we've showed so far, AFAICT, is that sampling is faster but produces a worse model. Why didn't those tests run until convergence?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14174][MLLIB] Accelerate KMeans via Min...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/11974#issuecomment-201701494
  
    **[Test build #54248 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54248/consoleFull)** for PR 11974 at commit [`b1dac54`](https://github.com/apache/spark/commit/b1dac5493d318a4e4a9ba9f3ac2a08aad50e4db0).
     * This patch **fails Scala style tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #69405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/69405/consoleFull)** for PR 11974 at commit [`83b5b8e`](https://github.com/apache/spark/commit/83b5b8e26b21a23a423a991564e28be24ed1551c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by zhengruifeng <gi...@git.apache.org>.
Github user zhengruifeng commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    @sethah In general, MiniBatchKMeans generate a worse model, but in many case the difference is small, while the speedup can be significant.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    **[Test build #77287 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/77287/testReport)** for PR 11974 at commit [`46d9c7b`](https://github.com/apache/spark/commit/46d9c7bbbfdd663a19212c2ddc7431ccd6293022).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #11974: [SPARK-14174][ML] Accelerate KMeans via Mini-Batch EM

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/11974
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/69405/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org