You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/05/25 06:59:57 UTC
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/13291
[Minor] [Doc] [ML] ml.clustering scala & python api doc sync
## What changes were proposed in this pull request?
Since we done Scala API audit for ml.clustering at #13148, we should also fix and update the corresponding Python API docs to keep them in sync.
## How was this patch tested?
Docs change, no tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-15361-followup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13291.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13291
----
commit e0bf84ebcc5e86cd58c196f3d0f4bd62dbe8f63f
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-05-25T06:52:24Z
ml.clustering scala & python api doc sync
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64699243
--- Diff: python/pyspark/ml/clustering.py ---
@@ -64,6 +64,21 @@ class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIte
.. note:: Experimental
GaussianMixture clustering.
+ This class performs expectation maximization for multivariate Gaussian
+ Mixture Models (GMMs). A GMM represents a composite distribution of
+ independent Gaussian distributions with associated "mixing" weights
+ specifying each's contribution to the composite.
+
+ Given a set of sample points, this class will maximize the log-likelihood
+ for a mixture of k Gaussians, iterating until the log-likelihood changes by
+ less than convergenceTol, or until it has reached the max number of iterations.
+ While this process is generally guaranteed to converge, it is not guaranteed
+ to find a global optimum.
+
+ Note: For high-dimensional data (with many features), this algorithm may perform poorly.
--- End diff --
super minor: This formats oddly in Sphinx - to match the scala doc format wise I think you could drop the indentation for the sentances under note, or if you wanted to do a PyDoc note call out you could use the `.. Note::` syntax
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64741733
--- Diff: python/pyspark/ml/clustering.py ---
@@ -227,15 +242,15 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
.. versionadded:: 1.5.0
"""
- k = Param(Params._dummy(), "k", "number of clusters to create",
+ k = Param(Params._dummy(), "k", "The number of clusters to create. Must be > 1.",
typeConverter=TypeConverters.toInt)
initMode = Param(Params._dummy(), "initMode",
- "the initialization algorithm. This can be either \"random\" to " +
+ "The initialization algorithm. This can be either \"random\" to " +
"choose random points as initial cluster centers, or \"k-means||\" " +
"to use a parallel variant of k-means++",
typeConverter=TypeConverters.toString)
- initSteps = Param(Params._dummy(), "initSteps", "steps for k-means initialization mode",
- typeConverter=TypeConverters.toInt)
+ initSteps = Param(Params._dummy(), "initSteps", "The number of steps for k-means|| " +
--- End diff --
We just sync the param docs rather than API docs. Actually the Python API docs for params are missing for all classes. I think we should add them in the next release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13291
**[Test build #59671 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59671/consoleFull)** for PR 13291 at commit [`e0bf84e`](https://github.com/apache/spark/commit/e0bf84ebcc5e86cd58c196f3d0f4bd62dbe8f63f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/13291
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13291
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64946619
--- Diff: python/pyspark/ml/clustering.py ---
@@ -64,6 +64,21 @@ class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIte
.. note:: Experimental
GaussianMixture clustering.
+ This class performs expectation maximization for multivariate Gaussian
+ Mixture Models (GMMs). A GMM represents a composite distribution of
+ independent Gaussian distributions with associated "mixing" weights
+ specifying each's contribution to the composite.
+
+ Given a set of sample points, this class will maximize the log-likelihood
+ for a mixture of k Gaussians, iterating until the log-likelihood changes by
+ less than convergenceTol, or until it has reached the max number of iterations.
+ While this process is generally guaranteed to converge, it is not guaranteed
+ to find a global optimum.
+
+ Note: For high-dimensional data (with many features), this algorithm may perform poorly.
--- End diff --
Sounds like a good plan, once my other doc change PRs are in I'll go through the PyDoc again and see if anything looks oddly formatted.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64699018
--- Diff: python/pyspark/ml/clustering.py ---
@@ -227,15 +242,15 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
.. versionadded:: 1.5.0
"""
- k = Param(Params._dummy(), "k", "number of clusters to create",
+ k = Param(Params._dummy(), "k", "The number of clusters to create. Must be > 1.",
typeConverter=TypeConverters.toInt)
initMode = Param(Params._dummy(), "initMode",
- "the initialization algorithm. This can be either \"random\" to " +
+ "The initialization algorithm. This can be either \"random\" to " +
"choose random points as initial cluster centers, or \"k-means||\" " +
"to use a parallel variant of k-means++",
typeConverter=TypeConverters.toString)
- initSteps = Param(Params._dummy(), "initSteps", "steps for k-means initialization mode",
- typeConverter=TypeConverters.toInt)
+ initSteps = Param(Params._dummy(), "initSteps", "The number of steps for k-means|| " +
--- End diff --
Since were copying this over might as well also include "his is an advanced setting -- the default of 5 is almost always enough." from the scala side?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64946896
--- Diff: python/pyspark/ml/clustering.py ---
@@ -227,15 +242,15 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
.. versionadded:: 1.5.0
"""
- k = Param(Params._dummy(), "k", "number of clusters to create",
+ k = Param(Params._dummy(), "k", "The number of clusters to create. Must be > 1.",
typeConverter=TypeConverters.toInt)
initMode = Param(Params._dummy(), "initMode",
- "the initialization algorithm. This can be either \"random\" to " +
+ "The initialization algorithm. This can be either \"random\" to " +
"choose random points as initial cluster centers, or \"k-means||\" " +
"to use a parallel variant of k-means++",
typeConverter=TypeConverters.toString)
- initSteps = Param(Params._dummy(), "initSteps", "steps for k-means initialization mode",
- typeConverter=TypeConverters.toInt)
+ initSteps = Param(Params._dummy(), "initSteps", "The number of steps for k-means|| " +
--- End diff --
interesting, I hadn't noticed the doc params were different than the scaladoc for these params (normally they are more in sync yes?).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13291
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59671/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64741801
--- Diff: python/pyspark/ml/clustering.py ---
@@ -380,11 +395,11 @@ class BisectingKMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIte
.. versionadded:: 2.0.0
"""
- k = Param(Params._dummy(), "k", "number of clusters to create",
+ k = Param(Params._dummy(), "k", "The desired number of leaf clusters. Must be > 1.",
--- End diff --
Ditto.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64741783
--- Diff: python/pyspark/ml/clustering.py ---
@@ -64,6 +64,21 @@ class GaussianMixture(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIte
.. note:: Experimental
GaussianMixture clustering.
+ This class performs expectation maximization for multivariate Gaussian
+ Mixture Models (GMMs). A GMM represents a composite distribution of
+ independent Gaussian distributions with associated "mixing" weights
+ specifying each's contribution to the composite.
+
+ Given a set of sample points, this class will maximize the log-likelihood
+ for a mixture of k Gaussians, iterating until the log-likelihood changes by
+ less than convergenceTol, or until it has reached the max number of iterations.
+ While this process is generally guaranteed to converge, it is not guaranteed
+ to find a global optimum.
+
+ Note: For high-dimensional data (with many features), this algorithm may perform poorly.
--- End diff --
Sounds good. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13291
**[Test build #59671 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59671/consoleFull)** for PR 13291 at commit [`e0bf84e`](https://github.com/apache/spark/commit/e0bf84ebcc5e86cd58c196f3d0f4bd62dbe8f63f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13291#issuecomment-221489999
**[Test build #59264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59264/consoleFull)** for PR 13291 at commit [`e0bf84e`](https://github.com/apache/spark/commit/e0bf84ebcc5e86cd58c196f3d0f4bd62dbe8f63f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13291#issuecomment-221491834
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59264/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/13291
Merged to master/branch-2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/13291#issuecomment-222452002
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & python api doc ...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/13291
jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/13291#issuecomment-221491831
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/13291#issuecomment-221491769
**[Test build #59264 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59264/consoleFull)** for PR 13291 at commit [`e0bf84e`](https://github.com/apache/spark/commit/e0bf84ebcc5e86cd58c196f3d0f4bd62dbe8f63f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [Minor] [Doc] [ML] ml.clustering scala & pytho...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/13291#discussion_r64699369
--- Diff: python/pyspark/ml/clustering.py ---
@@ -380,11 +395,11 @@ class BisectingKMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIte
.. versionadded:: 2.0.0
"""
- k = Param(Params._dummy(), "k", "number of clusters to create",
+ k = Param(Params._dummy(), "k", "The desired number of leaf clusters. Must be > 1.",
--- End diff --
Do we also maybe want to copy the full text here since we are updating it anyways? "The actual number could be smaller if there are no divisible leaf clusters."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org