You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by caneGuy <gi...@git.apache.org> on 2018/06/13 10:29:36 UTC
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
GitHub user caneGuy opened a pull request:
https://github.com/apache/spark/pull/21552
[SPARK-24544][SQL] Print actual failure cause when look up function failed
## What changes were proposed in this pull request?
When we operate as below:
`
0: jdbc:hive2://xxx/> create function funnel_analysis as 'com.xxx.hive.extend.udf.UapFunnelAnalysis';
`
`
0: jdbc:hive2://xxx/> select funnel_analysis(1,",",1,'');
Error: org.apache.spark.sql.AnalysisException: Undefined function: 'funnel_analysis'. This function is neither a registered temporary function nor a permanent function registered in the database 'xxx'.; line 1 pos 7 (state=,code=0)
`
`
0: jdbc:hive2://xxx/> describe function funnel_analysis;
+-----------------------------------------------------------+--+
| function_desc |
+-----------------------------------------------------------+--+
| Function: mifi.funnel_analysis |
| Class: com.xxx.hive.extend.udf.UapFunnelAnalysis |
| Usage: N/A. |
+-----------------------------------------------------------+--+
`
We can see describe funtion will get right information,but when we actually use this funtion,we will get an undefined exception.
Which is really misleading,the real cause is below:
`
No handler for Hive UDF 'com.xiaomi.mifi.hive.extend.udf.UapFunnelAnalysis': java.lang.IllegalStateException: Should not be called directly;
at org.apache.hadoop.hive.ql.udf.generic.GenericUDTF.initialize(GenericUDTF.java:72)
at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector$lzycompute(hiveUDFs.scala:204)
at org.apache.spark.sql.hive.HiveGenericUDTF.outputInspector(hiveUDFs.scala:204)
at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema$lzycompute(hiveUDFs.scala:212)
at org.apache.spark.sql.hive.HiveGenericUDTF.elementSchema(hiveUDFs.scala:212)
`
This patch print the actual failure for quick debugging.
## How was this patch tested?
UT
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/caneGuy/spark zhoukang/print-warning
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21552.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21552
----
commit 15d018a1e72ddf6cd29d6942359b9e6bd5547f67
Author: zhoukang <zh...@...>
Date: 2018-06-13T10:23:01Z
[SPARK][SQL] Print actual failure cause when look up function failed
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on the issue:
https://github.com/apache/spark/pull/21552
cc @cloud-fan Could help review this?Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #92926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92926/testReport)** for PR 21552 at commit [`70835b7`](https://github.com/apache/spark/commit/70835b7db99526d77c32fef72f59afe61c76174f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy closed the pull request at:
https://github.com/apache/spark/pull/21552
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by viirya <gi...@git.apache.org>.
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r200224239
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
Instead of logging warning here, shall we set up the cause for thrown `NoSuchFunctionException`?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92928/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r200157051
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
@maropu Any more suggestion?Thanks
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r200828333
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
@viirya Thanks, i will set up the cause for `NoSuchFunctionException ` later
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on the issue:
https://github.com/apache/spark/pull/21552
@maropu May be i will do this check?As @cloud-fan mentioned.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Can one of the admins verify this patch?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21552
@cloud-fan That is realted to https://github.com/apache/spark/pull/18527.
If you override `public StructObjectInspector initialize(StructObjectInspector argOIs)` instead of `public StructObjectInspector initialize(ObjectInspector[] argOIs)`, it throws IllegalStateException (HIVE-12377).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r201970972
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
I have update that set up the root cause for `NoSuchFunctionException`.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on the issue:
https://github.com/apache/spark/pull/21552
ah, I see. It seems we should do that.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r195894982
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
How about `logWarning(s"Failed to look up a function: $funcName")`? I feel printing `Utils.exceptionString(error)` is a little meaningless.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92926/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21552
Why do we call `GenericUDTF.initialize` during function lookup?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/91765/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #92926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92926/testReport)** for PR 21552 at commit [`70835b7`](https://github.com/apache/spark/commit/70835b7db99526d77c32fef72f59afe61c76174f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #91765 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91765/testReport)** for PR 21552 at commit [`15d018a`](https://github.com/apache/spark/commit/15d018a1e72ddf6cd29d6942359b9e6bd5547f67).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #92928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92928/testReport)** for PR 21552 at commit [`918f355`](https://github.com/apache/spark/commit/918f355e58e22f152a6158f45039827129c93c8f).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by caneGuy <gi...@git.apache.org>.
Github user caneGuy commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r196378408
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
Actually, this patch intended to print actual exception.As the description mentioned, this exception will helpful for troubleshooting.
Thanks @maropu
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/21552
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/21552
Why Spark doesn't fail when create this function?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #91765 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91765/testReport)** for PR 21552 at commit [`15d018a`](https://github.com/apache/spark/commit/15d018a1e72ddf6cd29d6942359b9e6bd5547f67).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #21552: [SPARK-24544][SQL] Print actual failure cause when look ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/21552
**[Test build #92928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92928/testReport)** for PR 21552 at commit [`918f355`](https://github.com/apache/spark/commit/918f355e58e22f152a6158f45039827129c93c8f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #21552: [SPARK-24544][SQL] Print actual failure cause whe...
Posted by maropu <gi...@git.apache.org>.
Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21552#discussion_r200215518
--- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveSessionCatalog.scala ---
@@ -131,6 +132,8 @@ private[sql] class HiveSessionCatalog(
Try(super.lookupFunction(funcName, children)) match {
case Success(expr) => expr
case Failure(error) =>
+ logWarning(s"Encounter a failure during looking up function:" +
+ s" ${Utils.exceptionString(error)}")
if (functionRegistry.functionExists(funcName)) {
--- End diff --
cc @gatorsmile @viirya
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org