You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2017/11/10 04:16:48 UTC
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/19713
[SPARK-22488] [SQL] Fix the view resolution issue in the SparkSession internal table() API
## What changes were proposed in this pull request?
The current internal `table()` API of `SparkSession` bypasses the Analyzer and directly calls `sessionState.catalog.lookupRelation` API. This skips the view resolution logics in our Analyzer rule `ResolveRelations`. This internal API is widely used by various DDL commands or the other internal APIs.
Users might get the strange error caused by view resolution when the default database is different.
```
Table or view not found: t1; line 1 pos 14
org.apache.spark.sql.AnalysisException: Table or view not found: t1; line 1 pos 14
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
```
This PR is to fix it by enforcing it to use `ResolveRelations` to resolve the table.
## How was this patch tested?
Added a test case and modified the existing test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark viewResolution
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19713.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19713
----
commit d87f33327b351cea493a065d144044cf2c1a069f
Author: gatorsmile <ga...@gmail.com>
Date: 2017-11-10T03:47:59Z
fix.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83678 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83678/testReport)** for PR 19713 at commit [`db2292f`](https://github.com/apache/spark/commit/db2292fe33525b8a9ed373101f54bfd62711089c).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19713
retest this please
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83674 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83674/testReport)** for PR 19713 at commit [`99936cc`](https://github.com/apache/spark/commit/99936ccd8cd5b127e67f4b32b4d7d3656c0cef78).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83664 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83664/testReport)** for PR 19713 at commit [`d87f333`](https://github.com/apache/spark/commit/d87f33327b351cea493a065d144044cf2c1a069f).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83721/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83678/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150384871
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -825,10 +825,11 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
spark.range(10).createOrReplaceTempView("tab1")
sql("ALTER TABLE tab1 RENAME TO tab2")
checkAnswer(spark.table("tab2"), spark.range(10).toDF())
- intercept[NoSuchTableException] { spark.table("tab1") }
+ val e = intercept[AnalysisException](spark.table("tab1")).getMessage
+ assert(e.contains("Table or view not found"))
--- End diff --
nit: add an empty line below this.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83674 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83674/testReport)** for PR 19713 at commit [`99936cc`](https://github.com/apache/spark/commit/99936ccd8cd5b127e67f4b32b4d7d3656c0cef78).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150173670
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -820,15 +820,16 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}
- test("rename temporary view") {
+ test("rename temporary table") {
--- End diff --
temporary view is more accurate?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83721/testReport)** for PR 19713 at commit [`c491974`](https://github.com/apache/spark/commit/c49197428168685ec0c3491ad7f16dd10f7e09e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150174211
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala ---
@@ -820,15 +820,16 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils {
}
}
- test("rename temporary view") {
+ test("rename temporary table") {
--- End diff --
Copied from 2.2 version... : )
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19713
thanks, merging to master! can you send a new PR for 2.2?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by felixcheung <gi...@git.apache.org>.
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19713
Sounds like we should have for 2.2.1
I’m looking at cutting 2.2.1 ASAP, could someone backport this and tag me?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83664 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83664/testReport)** for PR 19713 at commit [`d87f333`](https://github.com/apache/spark/commit/d87f33327b351cea493a065d144044cf2c1a069f).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83664/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83721 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83721/testReport)** for PR 19713 at commit [`c491974`](https://github.com/apache/spark/commit/c49197428168685ec0c3491ad7f16dd10f7e09e1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19713
it's a valid failure
```
Failed -------------------------------------------------------------------------
1. Failure: test cache, uncache and clearCache (@test_sparkSQL.R#735) ----------
error$message does not match "Error in uncacheTable : no such table - Table or view 'foo' not found in database 'default'".
Actual value: "Error in uncacheTable : analysis error - Table or view not found: foo;"
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by HyukjinKwon <gi...@git.apache.org>.
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150380582
--- Diff: R/pkg/tests/fulltests/test_sparkSQL.R ---
@@ -733,7 +733,7 @@ test_that("test cache, uncache and clearCache", {
expect_true(dropTempView("table1"))
expect_error(uncacheTable("foo"),
- "Error in uncacheTable : no such table - Table or view 'foo' not found in database 'default'")
+ "Error in uncacheTable : analysis error - Table or view not found: foo")
--- End diff --
tiny nit:
```r
expect_error(uncacheTable("foo"),
"Error in uncacheTable : analysis error - Table or view not found: foo")
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83717/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:
https://github.com/apache/spark/pull/19713
Hi, All.
It seems we missed `DROP TABLE IF EXISTS` cases here.
(I also forgot to check that during 2.2-backporting of this PR.)
This PR swallows `NoSuchTableException` and emits `AnalysisException`. `DROP TABLE IF EXISTS` expected `NoSuchTableException` in ddl.scala. It becomes a regression on 2.2.1.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83717/testReport)** for PR 19713 at commit [`c491974`](https://github.com/apache/spark/commit/c49197428168685ec0c3491ad7f16dd10f7e09e1).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83674/
Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83678 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83678/testReport)** for PR 19713 at commit [`db2292f`](https://github.com/apache/spark/commit/db2292fe33525b8a9ed373101f54bfd62711089c).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150174015
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ---
@@ -621,7 +622,7 @@ class SparkSession private(
}
private[sql] def table(tableIdent: TableIdentifier): DataFrame = {
--- End diff --
Yeah. many public APIs and DDL command processing are based on this internal API.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/19713#discussion_r150173854
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala ---
@@ -621,7 +622,7 @@ class SparkSession private(
}
private[sql] def table(tableIdent: TableIdentifier): DataFrame = {
--- End diff --
This is also called by the public `table` API, so this bug affect the public interface too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19713
cc @cloud-fan @jiangxb1987
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #19713: [SPARK-22488] [SQL] Fix the view resolution issue...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19713
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/19713
LGTM
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19713
Merged build finished. Test FAILed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #19713: [SPARK-22488] [SQL] Fix the view resolution issue in the...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19713
**[Test build #83717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83717/testReport)** for PR 19713 at commit [`c491974`](https://github.com/apache/spark/commit/c49197428168685ec0c3491ad7f16dd10f7e09e1).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org