You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by BryanCutler <gi...@git.apache.org> on 2018/05/25 17:22:15 UTC
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/21435
[SPARK-24392][PYTHON] Label pandas_udf as Experimental
## What changes were proposed in this pull request?
The pandas_udf functionality was introduced in 2.3.0, but is not completely stable and still evolving. This adds a label to indicate it is still an experimental API.
## How was this patch tested?
NA
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/BryanCutler/spark arrow-pandas_udf-experimental-SPARK-24392
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21435.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21435
----
commit 376149aa33b16f39f21fc209d11aef232faa2bbd
Author: Bryan Cutler <cu...@...>
Date: 2018-05-25T17:19:04Z
add experimental to pandas_udf
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91181/testReport)** for PR 21435 at commit [`fb05282`](https://github.com/apache/spark/commit/fb05282803f5c470c262263c2d5669f9a2f914db).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91197/testReport)** for PR 21435 at commit [`f977293`](https://github.com/apache/spark/commit/f977293d4b441b25dd00684571935f2894207645).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91209 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91209/testReport)** for PR 21435 at commit [`6b87330`](https://github.com/apache/spark/commit/6b873309b3db7022cfe47561c47d7ae320a46d65).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21435
I think this LGTM, should we add a note in migration doc from 2.3.0 to 2.3.1 for this change?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91181/testReport)** for PR 21435 at commit [`fb05282`](https://github.com/apache/spark/commit/fb05282803f5c470c262263c2d5669f9a2f914db).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21435
oh yea please.could we just make a section "Upgrading From Spark SQL 2.3.0 to 2.3.1 and above " here?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91163/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91196 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91196/testReport)** for PR 21435 at commit [`38a765d`](https://github.com/apache/spark/commit/38a765d96e723afe9d318098f6af540397664597).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191038814
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
+
--- End diff --
Anywhere is fine IMHO if it looks making sense. Another option is things like:
```
.. versionchanged:: 2.3
Arrow optimization added
.. note:: Experimental.
```
I just wanted to make sure this feature is experimental. If it's wrongly designed, we should consider a behaviour change as the last resort.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3608/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r190968782
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
+
--- End diff --
@BryanCutler, can we add a short node for createDataFrame and to Pandas with Pandas DataFrame input/output?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21435
I think it's just a mistake. It should have been done eariler. I think we should restore this righer description for this new feature before it's too late. It was a big new feature and still incomplete.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21435
Thanks @HyukjinKwon and @viirya !
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/21435
@HyukjinKwon's suggestion sounds good to me.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91163 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91163/testReport)** for PR 21435 at commit [`376149a`](https://github.com/apache/spark/commit/376149aa33b16f39f21fc209d11aef232faa2bbd).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191003569
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
+
--- End diff --
Yeah, I could have sworn that was added before but I don't see it anywhere. Would it make sense to add the note to the SQLconf description?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21435
I know this functionality was already released in 2.3.0 without this, but we have recently seen a number of bug reports indicating it is somewhat unstable and might need some more evolving. In that case, it might be better to mark it experimental now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91197/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91163 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91163/testReport)** for PR 21435 at commit [`376149a`](https://github.com/apache/spark/commit/376149aa33b16f39f21fc209d11aef232faa2bbd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191107787
--- Diff: docs/sql-programming-guide.md ---
@@ -1827,6 +1827,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see
- Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's ORC data source table and ORC vectorization would be applied. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.
- In version 2.3 and earlier, CSV rows are considered as malformed if at least one column value in the row is malformed. CSV parser dropped such rows in the DROPMALFORMED mode or outputs an error in the FAILFAST mode. Since Spark 2.4, CSV row is considered as malformed only when it contains malformed column values requested from CSV datasource, other values can be ignored. As an example, CSV file contains the "id,name" header and one row "1234". In Spark 2.4, selection of the id column consists of a row with one column value 1234 but in Spark 2.3 and earlier it is empty in the DROPMALFORMED mode. To restore the previous behavior, set `spark.sql.csv.parser.columnPruning.enabled` to `false`.
+## Upgrading From Spark SQL 2.3.0 to 2.3.1 and Above
--- End diff --
sure
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21435
I added a note in the migration guide for 2.3 to 2.4, but in branch-2.3 I don't see a section for 2.3.0 to 2.3.1. When backporting should I make that section for this or ok to just place at the end of 2.3?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191067913
--- Diff: docs/sql-programming-guide.md ---
@@ -1827,6 +1827,10 @@ working with timestamps in `pandas_udf`s to get the best performance, see
- Since Spark 2.0, Spark converts Parquet Hive tables by default for better performance. Since Spark 2.4, Spark converts ORC Hive tables by default, too. It means Spark uses its own ORC support by default instead of Hive SerDe. As an example, `CREATE TABLE t(id int) STORED AS ORC` would be handled with Hive SerDe in Spark 2.3, and in Spark 2.4, it would be converted into Spark's ORC data source table and ORC vectorization would be applied. To set `false` to `spark.sql.hive.convertMetastoreOrc` restores the previous behavior.
- In version 2.3 and earlier, CSV rows are considered as malformed if at least one column value in the row is malformed. CSV parser dropped such rows in the DROPMALFORMED mode or outputs an error in the FAILFAST mode. Since Spark 2.4, CSV row is considered as malformed only when it contains malformed column values requested from CSV datasource, other values can be ignored. As an example, CSV file contains the "id,name" header and one row "1234". In Spark 2.4, selection of the id column consists of a row with one column value 1234 but in Spark 2.3 and earlier it is empty in the DROPMALFORMED mode. To restore the previous behavior, set `spark.sql.csv.parser.columnPruning.enabled` to `false`.
+## Upgrading From Spark SQL 2.3.0 to 2.3.1 and Above
--- End diff --
sorry `Above` -> `above`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91198/testReport)** for PR 21435 at commit [`84b52f3`](https://github.com/apache/spark/commit/84b52f3f10afa2a9d5d386fd1090c1dcf993d59d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3617/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91198/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91193/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3626/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21435
yup Im seeing no harn to mtnetion it. It shouldn't be used in production for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3591/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21435
@gatorsmile @HyukjinKwon , I think we should probably make this change in light of some the recent issues, but I understand if we can not because it has already been released. What are your thoughts?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91181/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91196 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91196/testReport)** for PR 21435 at commit [`38a765d`](https://github.com/apache/spark/commit/38a765d96e723afe9d318098f6af540397664597).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191038840
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
+
--- End diff --
BTW, don't forget to update `group.apply()`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91196/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/21435
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3613/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91197/testReport)** for PR 21435 at commit [`f977293`](https://github.com/apache/spark/commit/f977293d4b441b25dd00684571935f2894207645).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/21435
Ahh, good idea thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21435
Merged to master and branch-2.3.
There were minor conflicts but I just resolved it by myself.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91198/testReport)** for PR 21435 at commit [`84b52f3`](https://github.com/apache/spark/commit/84b52f3f10afa2a9d5d386fd1090c1dcf993d59d).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91209/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91209 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91209/testReport)** for PR 21435 at commit [`6b87330`](https://github.com/apache/spark/commit/6b873309b3db7022cfe47561c47d7ae320a46d65).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191059323
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
--- End diff --
yeah, it was hard to find a consistent place to put it. I changed this one to be after the param definitions, before doctest. hopefully that's better.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/21435
Thanks @BryanCutler. still LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91193/testReport)** for PR 21435 at commit [`a1e70aa`](https://github.com/apache/spark/commit/a1e70aa04d698d8fef6d20da6cdfc5867d074462).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21435
**[Test build #91193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91193/testReport)** for PR 21435 at commit [`a1e70aa`](https://github.com/apache/spark/commit/a1e70aa04d698d8fef6d20da6cdfc5867d074462).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by BryanCutler <gi...@git.apache.org>.
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191042359
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
+
--- End diff --
Trying to add the note under `versionchanged` ends up with everything on one line and the note isn't highlighted like usual with Experimental, so I just made it it's own note.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/3616/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experimental
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21435
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21435: [SPARK-24392][PYTHON] Label pandas_udf as Experim...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/21435#discussion_r191043047
--- Diff: python/pyspark/sql/functions.py ---
@@ -2448,6 +2448,8 @@ def udf(f=None, returnType=StringType()):
@since(2.3)
def pandas_udf(f=None, returnType=None, functionType=None):
"""
+ .. note:: Experimental
--- End diff --
not a big deal but can we put this right before doctest just for consistency?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org