You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by zero323 <gi...@git.apache.org> on 2017/05/02 12:27:48 UTC
[GitHub] spark pull request #17831: [SPARK-18777] Return UDF from udf.register
GitHub user zero323 opened a pull request:
https://github.com/apache/spark/pull/17831
[SPARK-18777] Return UDF from udf.register
## What changes were proposed in this pull request?
- Move udf wrapping code from `functions.udf` to `functions.UserDefinedFunction`.
- Return wrapped udf from `catalog.registerFunction` and dependent methods.
- Update docstrings in `catalog.registerFunction` and `SQLContext.registerFunction`.
- Unit tests.
## How was this patch tested?
- Existing unit tests and docstests.
- Additional tests covering new feature.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zero323/spark SPARK-18777
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17831.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17831
----
commit 9edc5e939140e886959a4c48f59735036577578a
Author: zero323 <ze...@users.noreply.github.com>
Date: 2017-05-02T12:04:32Z
Return UDF from udf.register
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76416/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
LGTM, since this is just exposing existing functionality I think it makes sense to merge into 2.2 as well, but I'll leave this for a day incase anyone has any comments about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76381 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76381/testReport)** for PR 17831 at commit [`ced8814`](https://github.com/apache/spark/commit/ced8814a8a60bf3afaa791a8f7153ce53e7338ff).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17831
I want to know the UDF API change in more details and see if that blocks this PR. I would appreciate if I can check the details.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
I agree we need to be careful when making external API changes, but this is a pretty clear win for Python users and it matches the Scala API so I think improving this for Python users shouldn't necessarily block on any unification of the Scala API (is their a plan to improve this or JIRA associated with the planned Scala API changes I could subscribe to notifications on)?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
I don't think the Scala API is particularly consistent and I'd argue that this does actually match the current API as much as is reasonable. Looking at the udf.register interface defined in UDFRegistration.scala this does return either a `UserDefinedAggregateFunction` or `UserDefinedFunction` when passed in UserDefinedAggregateFunction or a Function0 through Function22. depending on which function is passed in most of the time. We do return Unit for the private `registerJavaFunction` & when registering a UDF1 through UDF22 . (Also @gatorsmile this doesn't change the scala API, its really just plumbing through the existing Scala API to Python).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17831
cc @viirya too who I believe is appropriate to review this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
@zero323 Will ping you when my PR is submitted. You can then add the UDF changes in Python.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76381 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76381/testReport)** for PR 17831 at commit [`ced8814`](https://github.com/apache/spark/commit/ced8814a8a60bf3afaa791a8f7153ce53e7338ff).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
Thank so for doing this, will let me cleanup some of my PySpark notebooks :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76382/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76381/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/17831
I think there is no conflict between #17848 and this. As of 2.2 we no longer return `UserDefinedFunction` from `udf` (and we never documented `UserDefinedFunctions`) so changes will have to be reflected in the arguments of `udf` and `UserDefinedFunction.__init__`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76383 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76383/testReport)** for PR 17831 at commit [`18c0253`](https://github.com/apache/spark/commit/18c025332318f547902888db981e5c7c25c1ea72).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17831#discussion_r114509033
--- Diff: python/pyspark/sql/catalog.py ---
@@ -237,23 +237,28 @@ def registerFunction(self, name, f, returnType=StringType()):
:param name: name of the UDF
:param f: python function
:param returnType: a :class:`pyspark.sql.types.DataType` object
+ :return: a wrapped :class:`UserDefinedFunction`
- >>> spark.catalog.registerFunction("stringLengthString", lambda x: len(x))
+ >>> strlen = spark.catalog.registerFunction("stringLengthString", len)
>>> spark.sql("SELECT stringLengthString('test')").collect()
[Row(stringLengthString(test)=u'4')]
+ >>> spark.sql("SELECT 'foo' AS text").select(strlen("text")).collect()
+ [Row(stringLengthString(text)=u'3')]
+
>>> from pyspark.sql.types import IntegerType
- >>> spark.catalog.registerFunction("stringLengthInt", lambda x: len(x), IntegerType())
+ >>> strlen = spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
--- End diff --
It looks `strlen` is not being used. Should we remove this or add an example for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76383/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76380/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76382/testReport)** for PR 17831 at commit [`e8615c2`](https://github.com/apache/spark/commit/e8615c20d004eee896cac81b794609015eb46e5d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/17831
Thanks everyone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17831
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
In Scala API, we have five different ways to register UDFs. We have to be more careful when changing the external APIs. It makes the users confused if we make the APIs more diverse. I do not suggest to make any further change in Python UDF, until we figure out how to make us easy to add more UDF types.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17831
Yea, thanks for chiming in. It helped me a lot to understand the context.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76380 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76380/testReport)** for PR 17831 at commit [`9edc5e9`](https://github.com/apache/spark/commit/9edc5e939140e886959a4c48f59735036577578a).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17831
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76382/testReport)** for PR 17831 at commit [`e8615c2`](https://github.com/apache/spark/commit/e8615c20d004eee896cac81b794609015eb46e5d).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
Thanks for making this PR with the details @gatorsmile it appears to be orthogonal to this change. Historically we've treated Python API parity fixes as closer to bug fixes rather than new features which is why I was thinking this made sense for 2.2, but I'll defer to your expertise in SQL and we can just target this for master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on a diff in the pull request:
https://github.com/apache/spark/pull/17831#discussion_r114516621
--- Diff: python/pyspark/sql/catalog.py ---
@@ -237,23 +237,28 @@ def registerFunction(self, name, f, returnType=StringType()):
:param name: name of the UDF
:param f: python function
:param returnType: a :class:`pyspark.sql.types.DataType` object
+ :return: a wrapped :class:`UserDefinedFunction`
- >>> spark.catalog.registerFunction("stringLengthString", lambda x: len(x))
+ >>> strlen = spark.catalog.registerFunction("stringLengthString", len)
>>> spark.sql("SELECT stringLengthString('test')").collect()
[Row(stringLengthString(test)=u'4')]
+ >>> spark.sql("SELECT 'foo' AS text").select(strlen("text")).collect()
+ [Row(stringLengthString(text)=u'3')]
+
>>> from pyspark.sql.types import IntegerType
- >>> spark.catalog.registerFunction("stringLengthInt", lambda x: len(x), IntegerType())
+ >>> strlen = spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
--- End diff --
Let's use `_`. It is a common to indicate discarded output and, unlike `repr`, it is portable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
JIRA: https://issues.apache.org/jira/browse/SPARK-20586 is opened.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/17831
This change LGTM.
I go to check #17848. It seems to me that the PR simply adds two flags into ScalaUDF. It appears that there is not API change regarding with existing UDF registration. I agreed with @holdenk and @HyukjinKwon that it is orthogonal to this change for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
So far, one of the biggest issues is we always assume UDF is deterministic. It could cause incorrect results after query optimization. I am working on a UDF API change to add more types for UDFs. Will submit a PR recently.
We can revisit this PR later.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76416 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76416/testReport)** for PR 17831 at commit [`bce80c4`](https://github.com/apache/spark/commit/bce80c413832738034224782227f73d5b9151625).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
@HyukjinKwon @holdenk Spark 2.2 RC has been out. This PR is not a bug fix. Thus, this PR will be merged to the master branch instead of 2.2. I will submit a PR to show the changes soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17831
Python interface `udf.register` needs to be consistent with the scala one.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76380 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76380/testReport)** for PR 17831 at commit [`9edc5e9`](https://github.com/apache/spark/commit/9edc5e939140e886959a4c48f59735036577578a).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/17831
@gatorsmile This sounds reasonable but I am not sure if I fully understand your concerns.
If anything this brings PySpark closer to the Scala API. At this moment we have
```
registerFunction(self, name: str, f: Callable[[T], U], returnType: DataType) -> None: ...
```
and we would move to:
```
registerFunction(self, name: str, f: Callable[[T], U], returnType: DataType) -> Callable[[Column, ...], Column]: ...
```
This, as pointed out by @holdenk, matches `register` API for `Function0` .. `Function22`.
If you're planning breaking changes in the Scala API, it may render this PR obsolete, but we don't commit here to any particular implementation. The only promise here is that registering udf for SQL applications, returns an object, which can be used with `DataFrame` API. I believe this sounds like a reasonable requirement for any upcoming API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17831#discussion_r114517172
--- Diff: python/pyspark/sql/catalog.py ---
@@ -237,23 +237,28 @@ def registerFunction(self, name, f, returnType=StringType()):
:param name: name of the UDF
:param f: python function
:param returnType: a :class:`pyspark.sql.types.DataType` object
+ :return: a wrapped :class:`UserDefinedFunction`
- >>> spark.catalog.registerFunction("stringLengthString", lambda x: len(x))
+ >>> strlen = spark.catalog.registerFunction("stringLengthString", len)
>>> spark.sql("SELECT stringLengthString('test')").collect()
[Row(stringLengthString(test)=u'4')]
+ >>> spark.sql("SELECT 'foo' AS text").select(strlen("text")).collect()
+ [Row(stringLengthString(text)=u'3')]
+
>>> from pyspark.sql.types import IntegerType
- >>> spark.catalog.registerFunction("stringLengthInt", lambda x: len(x), IntegerType())
+ >>> strlen = spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
--- End diff --
(Yea, discarded output is a better term ...)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/17831
It sounds orthogonal to me as well. LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/17831#discussion_r114515726
--- Diff: python/pyspark/sql/catalog.py ---
@@ -237,23 +237,28 @@ def registerFunction(self, name, f, returnType=StringType()):
:param name: name of the UDF
:param f: python function
:param returnType: a :class:`pyspark.sql.types.DataType` object
+ :return: a wrapped :class:`UserDefinedFunction`
- >>> spark.catalog.registerFunction("stringLengthString", lambda x: len(x))
+ >>> strlen = spark.catalog.registerFunction("stringLengthString", len)
>>> spark.sql("SELECT stringLengthString('test')").collect()
[Row(stringLengthString(test)=u'4')]
+ >>> spark.sql("SELECT 'foo' AS text").select(strlen("text")).collect()
+ [Row(stringLengthString(text)=u'3')]
+
>>> from pyspark.sql.types import IntegerType
- >>> spark.catalog.registerFunction("stringLengthInt", lambda x: len(x), IntegerType())
+ >>> strlen = spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
--- End diff --
Hm, I think capturing is rather better because it is what this API returns (and as it dose not return `None` anymore) if we don't have a better idea. Variable that indicates it's private, `_`, looks also fine to me. No strong opinion. I am fine if we don't have a simpler and cleaner idea.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.re...
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on a diff in the pull request:
https://github.com/apache/spark/pull/17831#discussion_r114511769
--- Diff: python/pyspark/sql/catalog.py ---
@@ -237,23 +237,28 @@ def registerFunction(self, name, f, returnType=StringType()):
:param name: name of the UDF
:param f: python function
:param returnType: a :class:`pyspark.sql.types.DataType` object
+ :return: a wrapped :class:`UserDefinedFunction`
- >>> spark.catalog.registerFunction("stringLengthString", lambda x: len(x))
+ >>> strlen = spark.catalog.registerFunction("stringLengthString", len)
>>> spark.sql("SELECT stringLengthString('test')").collect()
[Row(stringLengthString(test)=u'4')]
+ >>> spark.sql("SELECT 'foo' AS text").select(strlen("text")).collect()
+ [Row(stringLengthString(text)=u'3')]
+
>>> from pyspark.sql.types import IntegerType
- >>> spark.catalog.registerFunction("stringLengthInt", lambda x: len(x), IntegerType())
+ >>> strlen = spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
--- End diff --
We have to either capture the output (we could use `_` I guess) or use it in the doctests:
```
>>> spark.catalog.registerFunction("stringLengthInt", len, IntegerType())
<function len>
```
I am not sure if it makes sense to use it, unless we provide more diverse tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76416 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76416/testReport)** for PR 17831 at commit [`bce80c4`](https://github.com/apache/spark/commit/bce80c413832738034224782227f73d5b9151625).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by zero323 <gi...@git.apache.org>.
Github user zero323 commented on the issue:
https://github.com/apache/spark/pull/17831
Thanks @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
I feel like that's an unrelated challenge. I'm happy to see other improvements but I'm worried that we will hold up changes for things which aren't happening soon - is there a JIRA for these changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by holdenk <gi...@git.apache.org>.
Github user holdenk commented on the issue:
https://github.com/apache/spark/pull/17831
@gatorsmile want to know if you're ok with this going into master or if you still have concerns about this if its targeted to 2.3?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17831: [SPARK-18777][PYTHON][SQL] Return UDF from udf.register
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17831
**[Test build #76383 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76383/testReport)** for PR 17831 at commit [`18c0253`](https://github.com/apache/spark/commit/18c025332318f547902888db981e5c7c25c1ea72).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org