You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by holdenk <gi...@git.apache.org> on 2015/12/14 22:06:51 UTC
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
GitHub user holdenk opened a pull request:
https://github.com/apache/spark/pull/10298
[SPARK-12296][PYSPARK][MLLIB] Feature parity for pyspark mllib standard scaler model
Some methods are missing, such as ways to access the std, mean, etc. This PR is for feature parity for pyspark.mllib.feature.StandardScaler & StandardScalerModel.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/holdenk/spark SPARK-12296-feature-parity-pyspark-mllib-StandardScalerModel
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10298.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10298
----
commit 712ac0a59af30e09fb43f914aa35225249f770c9
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-14T20:28:03Z
Add properties for getting the std and mean on the StandardScalerModel
commit de3591f810fd140487d535375f26b1e9335caf2a
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-14T20:32:31Z
add withStd and withMean so people can check what kind of scaling the model was trained for
commit 8450e4628ce8bbd23f2cb6a72c2f62613b896c5e
Author: Holden Karau <ho...@us.ibm.com>
Date: 2015-12-14T20:59:20Z
Is targetted for 2.0.0
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/10298#discussion_r47892645
--- Diff: python/pyspark/mllib/feature.py ---
@@ -174,6 +174,38 @@ def setWithStd(self, withStd):
self.call("setWithStd", withStd)
return self
+ @property
+ @since('2.0.0')
+ def withStd(self):
+ """
+ Returns if the model scales the data to unit standard deviation.
+ """
+ return self.call("withStd")
+
+ @property
+ @since('2.0.0')
+ def withMean(self):
+ """
+ Returns if the model centers the data before scaling.
+ """
+ return self.call("withMean")
+
+ @property
+ @since('2.0.0')
+ def std(self):
+ """
+ Return the column standard deviation values. Only set if model was trained withStd.
+ """
+ return self.call("std")
+
+ @property
+ @since('2.0.0')
+ def mean(self):
--- End diff --
@holdenk quick question - if `withMean` or `withStd` is `false`, does this return `None` or `0.0`? It is worth adding a little test for this case at all?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-165581637
**[Test build #47949 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47949/consoleFull)** for PR 10298 at commit [`6b1740a`](https://github.com/apache/spark/commit/6b1740afb40c9a35d8976c21596bff7a9d59d648).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-164567160
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47678/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-164562517
**[Test build #47678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47678/consoleFull)** for PR 10298 at commit [`8450e46`](https://github.com/apache/spark/commit/8450e4628ce8bbd23f2cb6a72c2f62613b896c5e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-165585871
**[Test build #47949 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47949/consoleFull)** for PR 10298 at commit [`6b1740a`](https://github.com/apache/spark/commit/6b1740afb40c9a35d8976c21596bff7a9d59d648).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-164567156
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/10298
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-164566989
**[Test build #47678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/47678/consoleFull)** for PR 10298 at commit [`8450e46`](https://github.com/apache/spark/commit/8450e4628ce8bbd23f2cb6a72c2f62613b896c5e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-166544995
LGTM, merged. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-165586018
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/47949/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/10298#issuecomment-165586016
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-12296][PYSPARK][MLLIB] Feature parity f...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/10298#discussion_r47959777
--- Diff: python/pyspark/mllib/feature.py ---
@@ -174,6 +174,38 @@ def setWithStd(self, withStd):
self.call("setWithStd", withStd)
return self
+ @property
+ @since('2.0.0')
+ def withStd(self):
+ """
+ Returns if the model scales the data to unit standard deviation.
+ """
+ return self.call("withStd")
+
+ @property
+ @since('2.0.0')
+ def withMean(self):
+ """
+ Returns if the model centers the data before scaling.
+ """
+ return self.call("withMean")
+
+ @property
+ @since('2.0.0')
+ def std(self):
+ """
+ Return the column standard deviation values. Only set if model was trained withStd.
+ """
+ return self.call("std")
+
+ @property
+ @since('2.0.0')
+ def mean(self):
--- End diff --
Thats a good point, so it turns out the comment was a little off (currently we compute the mean and std regardless although there is a todo to skip computation when both are set to false - although I'm not sure that makes any sense from a producing a usable model point of view). I've updated the comment text.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org