You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sethah <gi...@git.apache.org> on 2016/02/06 01:19:37 UTC
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
GitHub user sethah opened a pull request:
https://github.com/apache/spark/pull/11099
[SPARK-13068][PYSPARK][ML]
This patch adds the following changes:
* Creates type specific subclasses of `pyspark.ml.param.Param` with specialized validation for each type. This mirrors the param subclasses in Scala.
* Adds an optional parameter `isValid` to the `Param` constructor which is a function that maps a parameter value to a Boolean. (e.g. `lambda value: value >= 0`)
* Adds `_validate` and `_convertAndValidate` functions to `Param` class which help with type conversions and validation when param values are set.
* Adds a `ParamValidators` class for factory validation methods, mirroring Scala.
* Changes all existing ml params to use the type specific subclasses.
* Adds unit tests to verify that parameter type conversion occurs.
I think this is the best way to provide robust type checking/conversion mechanisms in Python without resorting to a confusing public API and to support arbitrary future cases for params. Overall, this change aims to make the Python params more closely match the param implementations in Scala.
I would be happy to split this PR if it makes it more manageable, though I think the changes are intertwined.
Follow up work:
* Change the Param copy method to call `_set` instead of changing the param map directly. [SPARK-12944](https://issues.apache.org/jira/browse/SPARK-12944)
* Change all param setters to use private `_set` function instead of changing param map directly.
cc @yanboliang @holdenk @davies
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sethah/spark SPARK-13068
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11099.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11099
----
commit d4c3a1d591225a36eac55f865b297bd0d7385ecf
Author: sethah <se...@gmail.com>
Date: 2016-02-04T22:58:19Z
python param subclasses and validation functions
commit a6c8dbed5a98bf1a6744259bc98005d8e0891b35
Author: sethah <se...@gmail.com>
Date: 2016-02-04T23:55:02Z
cleaning up shared param codegen
commit 5b6ae3e149b5289262be6417a1b1c1c5ed37a094
Author: sethah <se...@gmail.com>
Date: 2016-02-05T00:52:01Z
some doc strings and cleanup
commit 2796e0fa08aad44636988e46f494433c0e78b3f8
Author: sethah <se...@gmail.com>
Date: 2016-02-05T19:31:04Z
adding tests
commit 30b052c2eb9b26723b3dda0f1ec45565112ecaf3
Author: sethah <se...@gmail.com>
Date: 2016-02-05T21:56:37Z
minor cleanup
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-183447276
**[Test build #51191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51191/consoleFull)** for PR 11099 at commit [`14c879f`](https://github.com/apache/spark/commit/14c879fe4858bd7e055f69f98dfd26d8e46334b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52114476
--- Diff: python/pyspark/ml/feature.py ---
@@ -57,8 +58,8 @@ class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
.. versionadded:: 1.4.0
"""
- threshold = Param(Params._dummy(), "threshold",
- "threshold in binary classification prediction, in range [0, 1]")
+ threshold = FloatParam(Params._dummy(), "threshold",
--- End diff --
Why did we remove the range information in this one?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-183428016
jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-182150212
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191324672
**[Test build #52325 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52325/consoleFull)** for PR 11099 at commit [`1adda2d`](https://github.com/apache/spark/commit/1adda2dbf6054860d183c1f7eeb2656973e836a6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-192012604
@sethah Thanks for the PR! I just made some high-level comments on the JIRA.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r53615612
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala ---
@@ -46,7 +46,7 @@ class PolynomialExpansion(override val uid: String)
* @group param
*/
val degree = new IntParam(this, "degree", "the polynomial degree to expand (>= 1)",
- ParamValidators.gt(1))
+ ParamValidators.gtEq(1))
--- End diff --
This has been done in https://github.com/apache/spark/pull/11216, and I think that would be merged shortly (also has a related test case addition).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52225995
--- Diff: python/pyspark/ml/param/__init__.py ---
@@ -49,11 +53,21 @@ def _copy_new_parent(self, parent):
else:
raise ValueError("Cannot copy from non-dummy parent %s." % parent)
+ def _validate(self, value):
+ if not self.isValid(value):
+ raise ValueError("{parent} parameter {name} given invalid value {value}"
+ .format(parent=self.parent, name=self.name, value=str(value)))
+
+ def _convertAndValidate(self, value):
--- End diff --
I added a docstring to the base classes. I am not sure if the derived classes need docstrings as well?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-181590812
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50939/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah closed the pull request at:
https://github.com/apache/spark/pull/11099
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52134728
--- Diff: python/pyspark/ml/feature.py ---
@@ -57,8 +58,8 @@ class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
.. versionadded:: 1.4.0
"""
- threshold = Param(Params._dummy(), "threshold",
- "threshold in binary classification prediction, in range [0, 1]")
+ threshold = FloatParam(Params._dummy(), "threshold",
--- End diff --
The range information was incorrect. There are Python unit tests that set this value to 2.0. Further, if you check the binarizer threshold param in Scala, there is no range validation check. I do believe there is no real range restriction on this parameter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-180643330
**[Test build #50851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50851/consoleFull)** for PR 11099 at commit [`30b052c`](https://github.com/apache/spark/commit/30b052c2eb9b26723b3dda0f1ec45565112ecaf3).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-195596930
I'm closing this PR and opening a new one to reflect the changes discussed on the Jira.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52399182
--- Diff: python/pyspark/ml/feature.py ---
@@ -57,8 +58,8 @@ class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
.. versionadded:: 1.4.0
"""
- threshold = Param(Params._dummy(), "threshold",
- "threshold in binary classification prediction, in range [0, 1]")
+ threshold = FloatParam(Params._dummy(), "threshold",
--- End diff --
You're correct, I didn't catch that. I have updated it to the correct description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191354532
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52329/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-181573760
**[Test build #50939 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50939/consoleFull)** for PR 11099 at commit [`27d0b7e`](https://github.com/apache/spark/commit/27d0b7ee422ec8348abe71ae72f75d2208a96d77).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-181590253
**[Test build #50939 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50939/consoleFull)** for PR 11099 at commit [`27d0b7e`](https://github.com/apache/spark/commit/27d0b7ee422ec8348abe71ae72f75d2208a96d77).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191339835
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-183432857
**[Test build #51191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51191/consoleFull)** for PR 11099 at commit [`14c879f`](https://github.com/apache/spark/commit/14c879fe4858bd7e055f69f98dfd26d8e46334b7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52114499
--- Diff: python/pyspark/ml/param/__init__.py ---
@@ -49,11 +53,21 @@ def _copy_new_parent(self, parent):
else:
raise ValueError("Cannot copy from non-dummy parent %s." % parent)
+ def _validate(self, value):
+ if not self.isValid(value):
+ raise ValueError("{parent} parameter {name} given invalid value {value}"
+ .format(parent=self.parent, name=self.name, value=str(value)))
+
+ def _convertAndValidate(self, value):
--- End diff --
While this is an internal function, it might still be useful to have some pydoc text for the expected behaviour for ourselves.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191326348
ping! @jkbradley - since you wrote `Param` validation in Scala, do you think you could weigh in on this change?
Perhaps @holdenk or @yanboliang could have a look as well?
Please note that this change is basically a pre-requisite to getting [SPARK-12944](https://issues.apache.org/jira/browse/SPARK-12944) (a bug) fixed since it is inherently an issue with how we validate param types in Pyspark.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-183447459
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r53809290
--- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/PolynomialExpansion.scala ---
@@ -46,7 +46,7 @@ class PolynomialExpansion(override val uid: String)
* @group param
*/
val degree = new IntParam(this, "degree", "the polynomial degree to expand (>= 1)",
- ParamValidators.gt(1))
+ ParamValidators.gtEq(1))
--- End diff --
Thanks for pointing that out, I'll remove this change in the next commit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-180708243
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-180708244
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50865/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-180635248
**[Test build #50851 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/50851/consoleFull)** for PR 11099 at commit [`30b052c`](https://github.com/apache/spark/commit/30b052c2eb9b26723b3dda0f1ec45565112ecaf3).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-183447460
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51191/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191354278
**[Test build #52329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52329/consoleFull)** for PR 11099 at commit [`16e9233`](https://github.com/apache/spark/commit/16e9233e78cb0360a94c92780017a38fdf9ca04c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191354528
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-182150214
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51015/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-181590809
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52398684
--- Diff: python/pyspark/ml/feature.py ---
@@ -57,8 +58,8 @@ class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
.. versionadded:: 1.4.0
"""
- threshold = Param(Params._dummy(), "threshold",
- "threshold in binary classification prediction, in range [0, 1]")
+ threshold = FloatParam(Params._dummy(), "threshold",
--- End diff --
Besides, I think this description is not correct too. In Scala, it is `threshold used to binarize continuous features`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191348742
**[Test build #52329 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52329/consoleFull)** for PR 11099 at commit [`16e9233`](https://github.com/apache/spark/commit/16e9233e78cb0360a94c92780017a38fdf9ca04c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191339676
**[Test build #52325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/52325/consoleFull)** for PR 11099 at commit [`1adda2d`](https://github.com/apache/spark/commit/1adda2dbf6054860d183c1f7eeb2656973e836a6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11099#discussion_r52135764
--- Diff: python/pyspark/ml/feature.py ---
@@ -57,8 +58,8 @@ class Binarizer(JavaTransformer, HasInputCol, HasOutputCol):
.. versionadded:: 1.4.0
"""
- threshold = Param(Params._dummy(), "threshold",
- "threshold in binary classification prediction, in range [0, 1]")
+ threshold = FloatParam(Params._dummy(), "threshold",
--- End diff --
Cool :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML] Extend Pyspark para...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-191339840
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/52325/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13068][PYSPARK][ML]
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11099#issuecomment-180643375
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/50851/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org