You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2016/01/31 10:42:44 UTC
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
GitHub user yanboliang opened a pull request:
https://github.com/apache/spark/pull/10999
[SPARK-13035] [ML] [PySpark] PySpark ml.clustering support export/import
PySpark ml.clustering support export/import.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yanboliang/spark spark-13035
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10999.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10999
----
commit dffafbf67782d4ee16524ad7a6edb4090e26c786
Author: Yanbo Liang <yb...@gmail.com>
Date: 2016-01-31T09:41:15Z
PySpark ml.clustering support export/import
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10999
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/10999#discussion_r51375854
--- Diff: python/pyspark/ml/clustering.py ---
@@ -69,6 +70,25 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+ >>> import os, tempfile
--- End diff --
hmm... Here we combine the test and example functions. I do not have strong preference about whether this should live here or in tests file. @jkbradley
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10999#issuecomment-177456319
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10999#issuecomment-177456287
**[Test build #50461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50461/consoleFull)** for PR 10999 at commit [`dffafbf`](https://github.com/apache/spark/commit/dffafbf67782d4ee16524ad7a6edb4090e26c786).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class KMeansModel(JavaModel, MLWritable, MLReadable):`
* `class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol, HasSeed,`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10999#discussion_r52688614
--- Diff: python/pyspark/ml/clustering.py ---
@@ -69,6 +70,25 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+ >>> import os, tempfile
--- End diff --
Sure thing will do
On Thursday, February 11, 2016, Xiangrui Meng <no...@github.com>
wrote:
> In python/pyspark/ml/clustering.py
> <https://github.com/apache/spark/pull/10999#discussion_r52688040>:
>
> > @@ -69,6 +70,25 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
> > True
> > >>> rows[2].prediction == rows[3].prediction
> > True
> > + >>> import os, tempfile
>
> I agree with @holdenk <https://github.com/holdenk> . This may be too
> verbose for a doctest. We can move the temp directory setup in test
> preparation (where we initialize sqlContext) and clean up. We can do that
> in a separate PR. @holdenk <https://github.com/holdenk> Could you create
> a JIRA for it? Thanks!
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/10999/files#r52688040>.
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10999#discussion_r51371337
--- Diff: python/pyspark/ml/clustering.py ---
@@ -69,6 +70,25 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+ >>> import os, tempfile
--- End diff --
this is maybe a bit much for a doctest since in general they are supposed to be example-ish. Maybe this should be in the tests file instead? Just a suggestion though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10999#issuecomment-177448887
**[Test build #50461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50461/consoleFull)** for PR 10999 at commit [`dffafbf`](https://github.com/apache/spark/commit/dffafbf67782d4ee16524ad7a6edb4090e26c786).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10999#issuecomment-177456320
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50461/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/10999#issuecomment-183117738
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13035] [ML] [PySpark] PySpark ml.cluste...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/10999#discussion_r52688040
--- Diff: python/pyspark/ml/clustering.py ---
@@ -69,6 +70,25 @@ class KMeans(JavaEstimator, HasFeaturesCol, HasPredictionCol, HasMaxIter, HasTol
True
>>> rows[2].prediction == rows[3].prediction
True
+ >>> import os, tempfile
--- End diff --
I agree with @holdenk . This may be too verbose for a doctest. We can move the temp directory setup in test preparation (where we initialize `sqlContext`) and clean up. We can do that in a separate PR. @holdenk Could you create a JIRA for it? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org