You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2018/01/04 23:59:50 UTC
[GitHub] spark pull request #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic p...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/20162
[SPARK-22965] [PySpark] [SQL] Add deterministic parameter to registerJavaFunction
## What changes were proposed in this pull request?
To register a JAVA UDF in PySpark, users are unable to specify the registered UDF is not deterministic. The proposal is to add the extra parameter deterministic at the end of the function registerJavaFunction
Below is an example.
```
>>> from pyspark.sql.types import DoubleType
>>> sqlContext.registerJavaFunction("javaRand",
... "test.org.apache.spark.sql.JavaRandUDF", DoubleType(), deterministic=False)
>>> sqlContext.sql("SELECT javaRand(3)").collect()
[Row(UDF:javaRand(3)=3.12345)]
```
## How was this patch tested?
Added test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark registerJava
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20162.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20162
----
commit cf0076b9257d65605ed3153f0b59cd89cdb145fc
Author: gatorsmile <ga...@...>
Date: 2018-01-04T23:57:51Z
fix
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85708 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85708/testReport)** for PR 20162 at commit [`7e4f3c0`](https://github.com/apache/spark/commit/7e4f3c0f0c4082bf166cec0e72f9f86f5d23aac8).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85707/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85708 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85708/testReport)** for PR 20162 at commit [`7e4f3c0`](https://github.com/apache/spark/commit/7e4f3c0f0c4082bf166cec0e72f9f86f5d23aac8).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85708/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85706 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85706/testReport)** for PR 20162 at commit [`cf0076b`](https://github.com/apache/spark/commit/cf0076b9257d65605ed3153f0b59cd89cdb145fc).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic p...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20162#discussion_r159785534
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -601,15 +601,32 @@ class UDFRegistration private[sql] (functionRegistry: FunctionRegistry) extends
//////////////////////////////////////////////////////////////////////////////////////////////
/**
- * Register a Java UDF class using reflection, for use from pyspark
+ * Register a deterministic Java UDF class using reflection, for use from pyspark
*
* @param name udf name
* @param className fully qualified class name of udf
* @param returnDataType return type of udf. If it is null, spark would try to infer
* via reflection.
*/
private[sql] def registerJava(name: String, className: String, returnDataType: DataType): Unit = {
--- End diff --
Although this is private and only used for PySpark, the JAVA users still could call it. For not breaking the existing behavior, I keep the original one unchanged. Also added a test case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85707/testReport)** for PR 20162 at commit [`d63e816`](https://github.com/apache/spark/commit/d63e816672da36266b81989ba6c07bb398aaaf4e).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85706/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85706 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85706/testReport)** for PR 20162 at commit [`cf0076b`](https://github.com/apache/spark/commit/cf0076b9257d65605ed3153f0b59cd89cdb145fc).
* This patch **fails Python style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20162
**[Test build #85707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85707/testReport)** for PR 20162 at commit [`d63e816`](https://github.com/apache/spark/commit/d63e816672da36266b81989ba6c07bb398aaaf4e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic p...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:
https://github.com/apache/spark/pull/20162
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20162: [SPARK-22965] [PySpark] [SQL] Add deterministic paramete...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20162
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org