You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maropu <gi...@git.apache.org> on 2017/04/21 03:09:31 UTC
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
GitHub user maropu opened a pull request:
https://github.com/apache/spark/pull/17712
[SPARK-20416][SQL] Print UDF names in EXPLAIN
## What changes were proposed in this pull request?
This pr added `withName` in `UserDefinedFunction` for printing UDF names in EXPLAIN
## How was this patch tested?
Added tests in `UDFSuite`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maropu/spark SPARK-20416
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17712.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17712
----
commit 90c516fb3a7163a7291eb72b058a82efee0e0c1e
Author: Takeshi Yamamuro <ya...@apache.org>
Date: 2017-04-21T02:57:34Z
Print UDF names in Dataset APIs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76807 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76807/testReport)** for PR 17712 at commit [`61bd96c`](https://github.com/apache/spark/commit/61bd96c4cfd0abe075cde097e33091db6d2052ef).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76013/testReport)** for PR 17712 at commit [`90c516f`](https://github.com/apache/spark/commit/90c516fb3a7163a7291eb72b058a82efee0e0c1e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76013/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17712
cc @gatorsmile
This is related to the deterministic thing you want to do?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/17712
Why use a map? That's super unstructured and easy to break ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76057/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17712
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76059/testReport)** for PR 17712 at commit [`dd182e4`](https://github.com/apache/spark/commit/dd182e4f1981305852041debed23324c5f689a47).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76027 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76027/testReport)** for PR 17712 at commit [`b08154e`](https://github.com/apache/spark/commit/b08154e18abc6cf0bae900cd73f28df83adcaee8).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76013/testReport)** for PR 17712 at commit [`90c516f`](https://github.com/apache/spark/commit/90c516fb3a7163a7291eb72b058a82efee0e0c1e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112800640
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -45,14 +45,33 @@ import org.apache.spark.sql.types.DataType
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
- inputTypes: Option[Seq[DataType]]) {
+ inputTypes: Option[Seq[DataType]],
+ name: Option[String]) {
+
+ // Optionally used for printing an UDF name in EXPLAIN
+ def withName(name: String): UserDefinedFunction = {
+ UserDefinedFunction(f, dataType, inputTypes, Option(name))
+ }
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil)))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
+ }
+}
+
+object UserDefinedFunction {
--- End diff --
also need an unapply function
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112648834
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
yea, but, the addition affects binary compatibility?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/17712
@gatorsmile WDYT? If this pr possibly merged, I still open; otherwise I'll close.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17712
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112638493
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
@rxin How about this fix? https://github.com/apache/spark/pull/17712/files#diff-989058ad6f2f8c863fff5e2e07eddaa0R72
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76053/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/17712
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76053/testReport)** for PR 17712 at commit [`8800c3b`](https://github.com/apache/spark/commit/8800c3b15048bad5926e2b2ed280b042fa5c9d47).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112801078
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -45,14 +45,33 @@ import org.apache.spark.sql.types.DataType
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
- inputTypes: Option[Seq[DataType]]) {
+ inputTypes: Option[Seq[DataType]],
+ name: Option[String]) {
+
+ // Optionally used for printing an UDF name in EXPLAIN
+ def withName(name: String): UserDefinedFunction = {
+ UserDefinedFunction(f, dataType, inputTypes, Option(name))
+ }
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil)))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
+ }
+}
+
+object UserDefinedFunction {
--- End diff --
oh, it seems we couldn't add `unapply` there because:
```
[error] /Users/maropu/IdeaProjects/spark/spark-master/sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala:45: method unapply is defined twic
e
[error] conflicting symbols both originated in file '/Users/maropu/IdeaProjects/spark/spark-master/sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFu
nction.scala'
[error] case class UserDefinedFunction protected[sql] (
[error] ^
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112692885
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,20 +47,31 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
- // Optionally used for printing UDF names in EXPLAIN
- private var nameOption: Option[String] = None
+ protected def name: Option[String] = None
- def withName(name: String): this.type = {
- this.nameOption = Option(name)
- this
- }
+ // Optionally used for printing UDF names in EXPLAIN
--- End diff --
"Optionally"? I'd assume the old "UDF" is the default unless the name is given.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112795922
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
okay, I'll recheck
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112692504
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,20 +47,31 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
- // Optionally used for printing UDF names in EXPLAIN
- private var nameOption: Option[String] = None
+ protected def name: Option[String] = None
- def withName(name: String): this.type = {
- this.nameOption = Option(name)
- this
- }
+ // Optionally used for printing UDF names in EXPLAIN
+ def withName(name: String): this.type =
+ UserDefinedFunctionWithName(Option(name), f, dataType, inputTypes)
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), nameOption))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
}
}
+
+/**
+ * A user-defined function with a function name.
+ *
+ * @since 2.3.0
+ */
+@InterfaceStability.Stable
+case class UserDefinedFunctionWithName protected[sql] (
+ override val name: Option[String],
--- End diff --
Can a `UserDefinedFunctionWithName` have **no** name? When?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76024/testReport)** for PR 17712 at commit [`9cececb`](https://github.com/apache/spark/commit/9cececbab693aaf2f102df151c4d7cf306e4b73d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76050/testReport)** for PR 17712 at commit [`5d797a9`](https://github.com/apache/spark/commit/5d797a97fa936a4534ea381f6aa3cfd5545310ce).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76059/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76027 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76027/testReport)** for PR 17712 at commit [`b08154e`](https://github.com/apache/spark/commit/b08154e18abc6cf0bae900cd73f28df83adcaee8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76025 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76025/testReport)** for PR 17712 at commit [`9ab4302`](https://github.com/apache/spark/commit/9ab430279b699088f68e3955d06f6031734b9ec0).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76807 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76807/testReport)** for PR 17712 at commit [`61bd96c`](https://github.com/apache/spark/commit/61bd96c4cfd0abe075cde097e33091db6d2052ef).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112643902
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
@rxin I tried to make this immutable though, IIUC this is no easy & simple way to do that... any idea?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17712
Yes! My PR has not been submitted due to my family issues. In addition to the name and deterministic flag, we have another two Scala UDF properties based on the existing Hive UDF types. Instead of adding them one by one, we plan to use a Map.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/17712
Aha, good! We already have a related JIRA ticket for that? I'ld like to leave this issue to it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76027/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76052/testReport)** for PR 17712 at commit [`96bc89d`](https://github.com/apache/spark/commit/96bc89d49456bcd00d863950dc0da1271153d186).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76026 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76026/testReport)** for PR 17712 at commit [`1a1d436`](https://github.com/apache/spark/commit/1a1d436643f532054ebe575e508b57e725260f8c).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76057/testReport)** for PR 17712 at commit [`dd182e4`](https://github.com/apache/spark/commit/dd182e4f1981305852041debed23324c5f689a47).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76026 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76026/testReport)** for PR 17712 at commit [`1a1d436`](https://github.com/apache/spark/commit/1a1d436643f532054ebe575e508b57e725260f8c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76025/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76807/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76799 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76799/testReport)** for PR 17712 at commit [`821f47d`](https://github.com/apache/spark/commit/821f47d2f8c47ccb747f16b00a3334e0c94c17a8).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76026/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76799/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76053 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76053/testReport)** for PR 17712 at commit [`8800c3b`](https://github.com/apache/spark/commit/8800c3b15048bad5926e2b2ed280b042fa5c9d47).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112634044
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
Beside @rxin's comment, I'd also add `name` as an input parameter with default value `None`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17712
Sorry for the delay, just submitted a PR for addressing the related issues. The PR fixed the issue using a different way. Could you review that PR?
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112638782
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
@jaceklaskowski sorry, but I missed your point. Could you give more more?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112634952
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
@@ -256,10 +256,12 @@ class UDFSuite extends QueryTest with SharedSQLContext {
val sparkPlan = spark.sessionState.executePlan(explain).executedPlan
sparkPlan.executeCollect().map(_.getString(0).trim).headOption.getOrElse("")
}
- val udf1 = "myUdf1"
- val udf2 = "myUdf2"
- spark.udf.register(udf1, (n: Int) => { n + 1 })
- spark.udf.register(udf2, (n: Int) => { n * 1 })
- assert(explainStr(sql("SELECT myUdf1(myUdf2(1))")).contains(s"UDF:$udf1(UDF:$udf2(1))"))
+ val udf1Name = "myUdf1"
+ val udf2Name = "myUdf2"
+ val udf1 = spark.udf.register(udf1Name, (n: Int) => { n + 1 })
+ val udf2 = spark.udf.register(udf2Name, (n: Int) => { n * 1 })
+ assert(explainStr(sql("SELECT myUdf1(myUdf2(1))")).contains(s"UDF:$udf1Name(UDF:$udf2Name(1))"))
+ assert(explainStr(spark.range(1).select(udf1(udf2(functions.lit(1)))))
+ .contains(s"UDF:$udf1Name(UDF:$udf2Name(1))"))
--- End diff --
The goal of the change was to make sure that the names are the same for SQL and Dataset "modes". The test should check it (even though it does it using the above two tests the last one should rather check equality of SQL's and Dataset's outputs).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112754224
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
it will be fine if we add an explicit apply method and unapply method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17712
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112800829
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -45,14 +45,33 @@ import org.apache.spark.sql.types.DataType
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
- inputTypes: Option[Seq[DataType]]) {
+ inputTypes: Option[Seq[DataType]],
+ name: Option[String]) {
+
+ // Optionally used for printing an UDF name in EXPLAIN
+ def withName(name: String): UserDefinedFunction = {
+ UserDefinedFunction(f, dataType, inputTypes, Option(name))
+ }
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil)))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
+ }
+}
+
+object UserDefinedFunction {
--- End diff --
ok. Is it okay to update the MiMa file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112622098
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
can we create a new instance instead so this is immutable?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76025 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76025/testReport)** for PR 17712 at commit [`9ab4302`](https://github.com/apache/spark/commit/9ab430279b699088f68e3955d06f6031734b9ec0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112803151
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -45,14 +45,33 @@ import org.apache.spark.sql.types.DataType
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
- inputTypes: Option[Seq[DataType]]) {
+ inputTypes: Option[Seq[DataType]],
+ name: Option[String]) {
+
+ // Optionally used for printing an UDF name in EXPLAIN
+ def withName(name: String): UserDefinedFunction = {
+ UserDefinedFunction(f, dataType, inputTypes, Option(name))
+ }
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil)))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
+ }
+}
+
+object UserDefinedFunction {
--- End diff --
for now, I'll revert it...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76052/testReport)** for PR 17712 at commit [`96bc89d`](https://github.com/apache/spark/commit/96bc89d49456bcd00d863950dc0da1271153d186).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112803097
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -45,14 +45,33 @@ import org.apache.spark.sql.types.DataType
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
- inputTypes: Option[Seq[DataType]]) {
+ inputTypes: Option[Seq[DataType]],
+ name: Option[String]) {
+
+ // Optionally used for printing an UDF name in EXPLAIN
+ def withName(name: String): UserDefinedFunction = {
+ UserDefinedFunction(f, dataType, inputTypes, Option(name))
+ }
/**
* Returns an expression that invokes the UDF, using the given arguments.
*
* @since 1.3.0
*/
def apply(exprs: Column*): Column = {
- Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil)))
+ Column(ScalaUDF(f, dataType, exprs.map(_.expr), inputTypes.getOrElse(Nil), name))
+ }
+}
+
+object UserDefinedFunction {
--- End diff --
ah ok - that sucks. that means this will break compatibility ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76050/testReport)** for PR 17712 at commit [`5d797a9`](https://github.com/apache/spark/commit/5d797a97fa936a4534ea381f6aa3cfd5545310ce).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112627226
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
okay, I'll fix
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76024/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76799 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76799/testReport)** for PR 17712 at commit [`821f47d`](https://github.com/apache/spark/commit/821f47d2f8c47ccb747f16b00a3334e0c94c17a8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76050/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76024/testReport)** for PR 17712 at commit [`9cececb`](https://github.com/apache/spark/commit/9cececbab693aaf2f102df151c4d7cf306e4b73d).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `case class UserDefinedFunctionWithName protected[sql] (`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/17712
cc: @gatorsmile
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by ueshin <gi...@git.apache.org>.
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112646366
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala ---
@@ -47,12 +47,20 @@ case class UserDefinedFunction protected[sql] (
dataType: DataType,
inputTypes: Option[Seq[DataType]]) {
+ // Optionally used for printing UDF names in EXPLAIN
+ private var nameOption: Option[String] = None
--- End diff --
@maropu I guess @jaceklaskowski wants to make this like:
```scala
case class UserDefinedFunction protected[sql] (
f: AnyRef,
dataType: DataType,
inputTypes: Option[Seq[DataType]],
name: Option[String] = None) {
...
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/76052/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17712
**[Test build #76059 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/76059/testReport)** for PR 17712 at commit [`dd182e4`](https://github.com/apache/spark/commit/dd182e4f1981305852041debed23324c5f689a47).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17712
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17712: [SPARK-20416][SQL] Print UDF names in EXPLAIN
Posted by jaceklaskowski <gi...@git.apache.org>.
Github user jaceklaskowski commented on a diff in the pull request:
https://github.com/apache/spark/pull/17712#discussion_r112634273
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/UDFSuite.scala ---
@@ -256,10 +256,12 @@ class UDFSuite extends QueryTest with SharedSQLContext {
val sparkPlan = spark.sessionState.executePlan(explain).executedPlan
sparkPlan.executeCollect().map(_.getString(0).trim).headOption.getOrElse("")
}
- val udf1 = "myUdf1"
- val udf2 = "myUdf2"
- spark.udf.register(udf1, (n: Int) => { n + 1 })
- spark.udf.register(udf2, (n: Int) => { n * 1 })
- assert(explainStr(sql("SELECT myUdf1(myUdf2(1))")).contains(s"UDF:$udf1(UDF:$udf2(1))"))
+ val udf1Name = "myUdf1"
+ val udf2Name = "myUdf2"
+ val udf1 = spark.udf.register(udf1Name, (n: Int) => { n + 1 })
--- End diff --
Remove `{` and `}` (as they're not needed)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org