You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by BryanCutler <gi...@git.apache.org> on 2016/09/15 20:19:30 UTC
[GitHub] spark pull request #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param va...
GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/15113
[SPARK-17508][PYSPARK][ML] PySpark treat Param values None same as not setting the Param
## What changes were proposed in this pull request?
In PySpark, if params are set with a value of `None` it will get converted by Py4J to a typed value and usually result in an error. For the case of string params, it will assign them a `null` value and lead to a `java.lang.NullPointerException`. From the user perspective, this is not the expected outcome and can be confusing.
This change causes PySpark to not set Params with a value of `None`, and they will be treated the same as not being set. If a Param is already set, then attempting to set a value of `None` will have no effect. This is mostly useful for a user that could be automating Param settings or wrapping classes that use Params.
## How was this patch tested?
Added new unit tests for setting a Param to `None` and ran existing PySpark-ml tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark pyspark-param-none-SPARK-17508
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/15113.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #15113
----
commit 53bd9d2e6c896fa48284ca7b79189e6ae3545f83
Author: Bryan Cutler <cu...@gmail.com>
Date: 2016-09-15T20:09:11Z
change setting of Param to ignore None values
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15113
Is this PR still needed?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/15113
@srowen Here is the change that would treat PySpark Params with a value of `None` the same as not setting the Param in the first place. I also think this makes sense from a Python perspective, and I'll copy some others to see if there is a consensus.
cc @jkbradley @yanboliang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15113
**[Test build #65458 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65458/consoleFull)** for PR 15113 at commit [`53bd9d2`](https://github.com/apache/spark/commit/53bd9d2e6c896fa48284ca7b79189e6ae3545f83).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15113
So I think our current handling of `None` is confusing, but I'd really like to see what the other committers have to say on this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/15113
Hi @ueshin , there isn't anymore work for this, just a decision if this behavior should change. I still think it should and would like to hear some more opinions before I close this. Any thoughts @holdenk @jkbradley ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15113
**[Test build #65458 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65458/consoleFull)** for PR 15113 at commit [`53bd9d2`](https://github.com/apache/spark/commit/53bd9d2e6c896fa48284ca7b79189e6ae3545f83).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/15113
@yanboliang : I've certainly seen even Spark developers be tripped up with the handling of `None` by the Py4J bridge so I wouldn't be surprised if this is a more common problem than we might think.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15113
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65458/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15113
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/15113
I still think this makes sense, but maybe I'm the minority. I'll go ahead and close it unless anyone else thinks it should be changed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/15113
Hi @yanboliang , thanks for taking a look!
I'm more inclined to think that most Python users would think that `weightCol=None` does not cause a problem, as in this corresponding JIRA. Most are unaware that Py4J will convert the value to `null` behind the scenes. Let me show the LogisticRegression constructor as an example
```
@keyword_only
def __init__(self, featuresCol="features", labelCol="label", predictionCol="prediction",
maxIter=100, regParam=0.0, elasticNetParam=0.0, tol=1e-6, fitIntercept=True,
threshold=0.5, thresholds=None, probabilityCol="probability",
rawPredictionCol="rawPrediction", standardization=True, weightCol=None,
aggregationDepth=2):
```
Here it shows `maxIter=100` which means that the param `maxIter` has a default value of 100. Then it shows `weightCol=None`, but this does not mean that `weightCol` has a default value of `null`, it means that the param `weightCol` is not set by default. So why would it have a different effect if the user explicitly writes it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/15113
@BryanCutler Hi, are you still working on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/15113
@BryanCutler Thanks for working on this. I'm a bit worried that if users set ```weightCol = None``` for Python means he would like to set ```weightCol = null``` for Scala.
The common scenario would be:
```Scala
LogisticRegression(maxIter=5, regParam=0.0, weightCol="weight")
LogisticRegression(maxIter=5, regParam=0.0, weightCol="")
LogisticRegression(maxIter=5, regParam=0.0)
```
And users are unlikely to write the following code:
```
LogisticRegression(maxIter=5, regParam=0.0, weightCol=None)
```
However, it users actually do this, I'm more likely to consider that he will forcibly specify ```weightCol``` as ```null```. So I vote to not change this behavior. Thanks.
cc @srowen @MLnick @jkbradley
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param values No...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the issue:
https://github.com/apache/spark/pull/15113
Further more, ```weightCol=None``` in the Python API doc may be confused for users, I think we can add some annotations to clarify the meanings. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #15113: [SPARK-17508][PYSPARK][ML] PySpark treat Param va...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler closed the pull request at:
https://github.com/apache/spark/pull/15113
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org