You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2016/05/04 02:35:52 UTC
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/12885
[SPARK-15108] [SQL] Describe Permanent UDTF
#### What changes were proposed in this pull request?
When Describe a UDTF, the command returns a wrong result. The command is unable to find the function, which has been created and cataloged in the catalog but not in the functionRegistry.
This PR is to correct it. If the function is not in the functionRegistry, we will check the catalog for collecting the information of the UDTF function.
#### How was this patch tested?
Added test cases to verify the results
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark showFunction
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12885.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12885
----
commit ed0fc89b1708c99896917edb3a50212fc2b57fe0
Author: gatorsmile <ga...@gmail.com>
Date: 2016-05-04T02:24:59Z
fix
commit 241ff6f187da4ed63a7c2570ad0f4413e4f48257
Author: gatorsmile <ga...@gmail.com>
Date: 2016-05-04T02:25:50Z
Merge remote-tracking branch 'upstream/master' into showFunction
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217525623
LGTM. Merging to master and branch 2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216736531
cc @andrewor14 @yhuai @viirya
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12885#discussion_r62371164
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -115,22 +116,23 @@ case class DescribeFunction(
Row(s"Function: case") ::
Row(s"Usage: CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END - " +
s"When a = b, returns c; when a = d, return e; else return f") :: Nil
- case _ => sparkSession.sessionState.functionRegistry.lookupFunction(functionName) match {
- case Some(info) =>
+ case _ =>
+ try {
+ val info = sparkSession.sessionState.catalog.lookupFunctionInfo(functionName)
--- End diff --
nvm. Other changes look also needed. Let me take another look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216777847
Besides, when a function is not loaded, I remember in Hive you can't find it with describe command too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217055817
I do not think we need to do anything when we switch the database.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216736360
**[Test build #57705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57705/consoleFull)** for PR 12885 at commit [`241ff6f`](https://github.com/apache/spark/commit/241ff6f187da4ed63a7c2570ad0f4413e4f48257).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217235372
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57899/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217235370
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217504126
cc @yhuai Let me know if you have any other suggestion. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217525644
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217212500
**[Test build #57901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57901/consoleFull)** for PR 12885 at commit [`d4d12f5`](https://github.com/apache/spark/commit/d4d12f5d6274089c26dbd4e7b659b51c983dc716).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217235973
**[Test build #57901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57901/consoleFull)** for PR 12885 at commit [`d4d12f5`](https://github.com/apache/spark/commit/d4d12f5d6274089c26dbd4e7b659b51c983dc716).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216936789
First, we do not refresh `FunctionRegistry` when we switch the database. Thus, the current `FunctionRegistry` is a global registry. `FunctionRegistry` need a concept of database.
Second, when the function is loaded, `FunctionRegistry` has an entry `dbName.funcName`. This is not a clean way.
Third, I need to confirm what is the behavior of Hive. If what you said is right, this looks strange to me. From users' perspective, they create a function but they are unable to describe it to verify if the function exists or not. They have to call the function for loading the function. This looks strange to me. Some function call could be very expensive. This design does not make sense to me.
Forth, I think the users are allowed to describe a function without specifying the database name. At least, the parser allows it. If necessary, we can add the current database name when checking `FunctionRegistry`. So far, this is missing
Thanks for your inputs!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216736452
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57705/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216758934
: ) Partially right. Even if the function is loaded, we might not be able to find it. The reason is we might not specify the database name in `Describe Function`.
The current implementation of `FunctionRegistry` is not perfect. It misses a critical part. We need to introduce the concept of `database` there. I plan to submit another PR later for enhancing it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217045403
Do we need to refresh `FunctionRegistry` when we switch the database? I think `FunctionRegistry` is like a registry for functions in current session. At least in Hive the function registration works like that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12885#discussion_r62150647
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -129,7 +129,16 @@ case class DescribeFunction(
result
}
- case None => Seq(Row(s"Function: $functionName not found."))
+ case None =>
+ val currentDb = sparkSession.sessionState.catalog.getCurrentDatabase
+ if (sparkSession.externalCatalog.functionExists(currentDb, functionName)) {
+ val catalogFunction = sparkSession.externalCatalog.getFunction(currentDb, functionName)
+ Row(s"Function: ${catalogFunction.identifier}") ::
+ Row(s"Class: ${catalogFunction.className}") ::
+ Row(s"Usage: N/A") :: Nil
+ } else {
+ Seq(Row(s"Function: $functionName not found."))
+ }
--- End diff --
We also need to change the parser and change the interface of `DescribeFunction` to:
```scala
case class DescribeFunction(
functionName: FunctionIdentifier,
isExtended: Boolean) extends RunnableCommand {
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217236274
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57901/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217528932
Thank you!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216772123
Does `FunctionRegistry` need to consider database? I think it is just a registry for functions in current database?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12885#discussion_r62370858
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -115,22 +116,23 @@ case class DescribeFunction(
Row(s"Function: case") ::
Row(s"Usage: CASE a WHEN b THEN c [WHEN d THEN e]* [ELSE f] END - " +
s"When a = b, returns c; when a = d, return e; else return f") :: Nil
- case _ => sparkSession.sessionState.functionRegistry.lookupFunction(functionName) match {
- case Some(info) =>
+ case _ =>
+ try {
+ val info = sparkSession.sessionState.catalog.lookupFunctionInfo(functionName)
--- End diff --
Looks like this line is the only change that we need?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217236271
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216742634
When we can't find the function in functionRegistry, does it mean it is not loaded yet?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216771430
yea. But I remember in Hive you can't find the function when you don't specify database name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217235125
**[Test build #57899 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57899/consoleFull)** for PR 12885 at commit [`24b459c`](https://github.com/apache/spark/commit/24b459c032bd7a60abc6d77cbc13be5f0cd1d818).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class NoSuchPermanentFunctionException(db: String, func: String)`
* `class NoSuchFunctionException(db: String, func: String)`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12885#discussion_r62141717
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -129,7 +129,16 @@ case class DescribeFunction(
result
}
- case None => Seq(Row(s"Function: $functionName not found."))
+ case None =>
+ val currentDb = sparkSession.sessionState.catalog.getCurrentDatabase
+ if (sparkSession.externalCatalog.functionExists(currentDb, functionName)) {
+ val catalogFunction = sparkSession.externalCatalog.getFunction(currentDb, functionName)
+ Row(s"Function: ${catalogFunction.identifier}") ::
+ Row(s"Class: ${catalogFunction.className}") ::
+ Row(s"Usage: N/A") :: Nil
+ } else {
+ Seq(Row(s"Function: $functionName not found."))
+ }
--- End diff --
: ) Yeah. We can do that in your way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216726692
**[Test build #57705 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57705/consoleFull)** for PR 12885 at commit [`241ff6f`](https://github.com/apache/spark/commit/241ff6f187da4ed63a7c2570ad0f4413e4f48257).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-216736451
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by yhuai <gi...@git.apache.org>.
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/12885#discussion_r62130890
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/functions.scala ---
@@ -129,7 +129,16 @@ case class DescribeFunction(
result
}
- case None => Seq(Row(s"Function: $functionName not found."))
+ case None =>
+ val currentDb = sparkSession.sessionState.catalog.getCurrentDatabase
+ if (sparkSession.externalCatalog.functionExists(currentDb, functionName)) {
+ val catalogFunction = sparkSession.externalCatalog.getFunction(currentDb, functionName)
+ Row(s"Function: ${catalogFunction.identifier}") ::
+ Row(s"Class: ${catalogFunction.className}") ::
+ Row(s"Usage: N/A") :: Nil
+ } else {
+ Seq(Row(s"Function: $functionName not found."))
+ }
--- End diff --
Looks like we only need to change https://github.com/apache/spark/pull/12885/files#diff-d2a203f08c862bd762e6740c16e972f7R118 to `sparkSession.sessionState.catalog.lookupFunctionInfo`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12885#issuecomment-217211279
**[Test build #57899 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57899/consoleFull)** for PR 12885 at commit [`24b459c`](https://github.com/apache/spark/commit/24b459c032bd7a60abc6d77cbc13be5f0cd1d818).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-15108] [SQL] Describe Permanent UDTF
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12885
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org