You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by cloud-fan <gi...@git.apache.org> on 2018/01/03 14:42:57 UTC
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
GitHub user cloud-fan opened a pull request:
https://github.com/apache/spark/pull/20141
[SPARK-22945][SQL] add java UDF APIs in the functions object
## What changes were proposed in this pull request?
Currently Scala users can use UDF like
```
val foo = udf((i: Int) => Math.random() + i).asNondeterministic
df.select(foo('a))
```
Python users can also do it with similar APIs. However Java users can't do it, we should add Java UDF APIs in the functions object.
## How was this patch tested?
new tests
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/cloud-fan/spark udf
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20141.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20141
----
commit 160cb61af5be56ae16b92e8fea860821e60f0293
Author: Wenchen Fan <we...@...>
Date: 2018-01-03T12:27:57Z
add java UDF APIs in the functions object
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20141
thanks, merging to master/2.3
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20141
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85658/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/20141
cc @rxin @gatorsmile
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20141#discussion_r159579701
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3461,13 +3485,172 @@ object functions {
if (nullable) udf else udf.asNonNullable()
}
+ //////////////////////////////////////////////////////////////////////////////////////////////
+ // Java UDF functions
+ //////////////////////////////////////////////////////////////////////////////////////////////
+
+ /**
+ * Defines a Java UDF0 instance as user-defined function (UDF).
+ * The caller must specify the output data type, and there is no automatic input type coercion.
+ * By default the returned UDF is deterministic, to change it to nondeterministic, call the
+ * API `UserDefinedFunction.asNondeterministic()`.
+ *
+ * @group udf_funcs
+ * @since 2.3.0
+ */
+ def udf(f: UDF0[_], returnType: DataType): UserDefinedFunction = {
+ val func = f.asInstanceOf[UDF0[Any]].call()
+ UserDefinedFunction(() => func, returnType, inputTypes = None)
+ }
+
+ /**
+ * Defines a Java UDF1 instance as user-defined function (UDF).
+ * The caller must specify the output data type, and there is no automatic input type coercion.
+ * By default the returned UDF is deterministic, to change it to nondeterministic, call the
--- End diff --
Nit: `, to` -> `. To`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20141#discussion_r159439756
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -66,6 +66,7 @@ case class UserDefinedFunction protected[sql] (
*
* @since 1.3.0
*/
+ @scala.annotation.varargs
--- End diff --
to make it java friendly, or shall we create a new method for java?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/20141
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20141
**[Test build #85658 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85658/testReport)** for PR 20141 at commit [`711c027`](https://github.com/apache/spark/commit/711c0278a6d77472761414692c987d256293467e).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20141
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20141
**[Test build #85658 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85658/testReport)** for PR 20141 at commit [`711c027`](https://github.com/apache/spark/commit/711c0278a6d77472761414692c987d256293467e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20141#discussion_r159439981
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -3254,42 +3254,66 @@ object functions {
*/
def map_values(e: Column): Column = withExpr { MapValues(e.expr) }
- //////////////////////////////////////////////////////////////////////////////////////////////
- //////////////////////////////////////////////////////////////////////////////////////////////
-
// scalastyle:off line.size.limit
// scalastyle:off parameter.number
/* Use the following code to generate:
- (0 to 10).map { x =>
+
+ (0 to 10).foreach { x =>
val types = (1 to x).foldRight("RT")((i, s) => {s"A$i, $s"})
val typeTags = (1 to x).map(i => s"A$i: TypeTag").foldLeft("RT: TypeTag")(_ + ", " + _)
val inputTypes = (1 to x).foldRight("Nil")((i, s) => {s"ScalaReflection.schemaFor(typeTag[A$i]).dataType :: $s"})
println(s"""
- /**
- * Defines a deterministic user-defined function of ${x} arguments as user-defined
- * function (UDF). The data types are automatically inferred based on the function's
- * signature. To change a UDF to nondeterministic, call the API
- * `UserDefinedFunction.asNondeterministic()`.
- *
- * @group udf_funcs
- * @since 1.3.0
- */
- def udf[$typeTags](f: Function$x[$types]): UserDefinedFunction = {
- val ScalaReflection.Schema(dataType, nullable) = ScalaReflection.schemaFor[RT]
- val inputTypes = Try($inputTypes).toOption
- val udf = UserDefinedFunction(f, dataType, inputTypes)
- if (nullable) udf else udf.asNonNullable()
- }""")
+ |/**
+ | * Defines a Scala closure of $x arguments as user-defined function (UDF).
+ | * The data types are automatically inferred based on the Scala closure's
+ | * signature. By default the returned UDF is deterministic, to change it to
+ | * nondeterministic, call the API `UserDefinedFunction.asNondeterministic()`.
+ | *
+ | * @group udf_funcs
+ | * @since 1.3.0
+ | */
+ |def udf[$typeTags](f: Function$x[$types]): UserDefinedFunction = {
+ | val ScalaReflection.Schema(dataType, nullable) = ScalaReflection.schemaFor[RT]
+ | val inputTypes = Try($inputTypes).toOption
+ | val udf = UserDefinedFunction(f, dataType, inputTypes)
+ | if (nullable) udf else udf.asNonNullable()
+ |}""".stripMargin)
+ }
+
+ (0 to 10).foreach { i =>
+ val extTypeArgs = (0 to i).map(_ => "_").mkString(", ")
+ val anyTypeArgs = (0 to i).map(_ => "Any").mkString(", ")
+ val anyCast = s".asInstanceOf[UDF$i[$anyTypeArgs]]"
+ val anyParams = (1 to i).map(_ => "_: Any").mkString(", ")
+ val funcCall = if (i == 0) "() => func" else "func"
+ println(s"""
+ |/**
+ | * Defines a Java UDF$i instance as user-defined function (UDF).
+ | * The caller must specify the output data type, and there is no automatic input type coercion.
+ | * By default the returned UDF is deterministic, to change it to nondeterministic, call the
+ | * API `UserDefinedFunction.asNondeterministic()`.
+ | *
+ | * @group udf_funcs
+ | * @since 2.3.0
+ | */
+ |def udf(f: UDF$i[$extTypeArgs], returnType: DataType): UserDefinedFunction = {
+ | val func = f$anyCast.call($anyParams)
+ | UserDefinedFunction($funcCall, returnType, inputTypes = None)
+ |}""".stripMargin)
}
*/
+ //////////////////////////////////////////////////////////////////////////////////////////////
+ // Scala UDF functions
+ //////////////////////////////////////////////////////////////////////////////////////////////
+
/**
- * Defines a deterministic user-defined function of 0 arguments as user-defined
- * function (UDF). The data types are automatically inferred based on the function's
- * signature. To change a UDF to nondeterministic, call the API
- * `UserDefinedFunction.asNondeterministic()`.
+ * Defines a Scala closure of 0 arguments as user-defined function (UDF).
--- End diff --
to make the document consistent with `UDFRegistration.register`
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/20141
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20141
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/20141
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85629/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #20141: [SPARK-22945][SQL] add java UDF APIs in the funct...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/20141#discussion_r159439570
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/UDFRegistration.scala ---
@@ -110,29 +110,29 @@ class UDFRegistration private[sql] (functionRegistry: FunctionRegistry) extends
/* register 0-22 were generated by this script
- (0 to 22).map { x =>
+ (0 to 22).foreach { x =>
val types = (1 to x).foldRight("RT")((i, s) => {s"A$i, $s"})
- val typeTags = (1 to x).map(i => s"A${i}: TypeTag").foldLeft("RT: TypeTag")(_ + ", " + _)
+ val typeTags = (1 to x).map(i => s"A$i: TypeTag").foldLeft("RT: TypeTag")(_ + ", " + _)
val inputTypes = (1 to x).foldRight("Nil")((i, s) => {s"ScalaReflection.schemaFor[A$i].dataType :: $s"})
println(s"""
- /**
- * Registers a deterministic Scala closure of ${x} arguments as user-defined function (UDF).
- * @tparam RT return type of UDF.
- * @since 1.3.0
- */
- def register[$typeTags](name: String, func: Function$x[$types]): UserDefinedFunction = {
- val ScalaReflection.Schema(dataType, nullable) = ScalaReflection.schemaFor[RT]
- val inputTypes = Try($inputTypes).toOption
- def builder(e: Seq[Expression]) = if (e.length == $x) {
- ScalaUDF(func, dataType, e, inputTypes.getOrElse(Nil), Some(name), nullable, udfDeterministic = true)
- } else {
- throw new AnalysisException("Invalid number of arguments for function " + name +
- ". Expected: $x; Found: " + e.length)
- }
- functionRegistry.createOrReplaceTempFunction(name, builder)
- val udf = UserDefinedFunction(func, dataType, inputTypes).withName(name)
- if (nullable) udf else udf.asNonNullable()
- }""")
+ |/**
--- End diff --
just code style change, to use the multiline string format.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20141
**[Test build #85629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85629/testReport)** for PR 20141 at commit [`160cb61`](https://github.com/apache/spark/commit/160cb61af5be56ae16b92e8fea860821e60f0293).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #20141: [SPARK-22945][SQL] add java UDF APIs in the functions ob...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/20141
**[Test build #85629 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85629/testReport)** for PR 20141 at commit [`160cb61`](https://github.com/apache/spark/commit/160cb61af5be56ae16b92e8fea860821e60f0293).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org