You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by skambha <gi...@git.apache.org> on 2017/02/25 23:55:11 UTC
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
GitHub user skambha opened a pull request:
https://github.com/apache/spark/pull/17067
[SPARK-19602][SQL][TESTS] Add tests for qualified column names
## What changes were proposed in this pull request?
- Add tests covering different scenarios with qualified column names without any code changes.
- Please see Section 2 in the design doc for the various test scenarios [here](https://issues.apache.org/jira/secure/attachment/12854681/Design_ColResolution_JIRA19602.pdf)
- As part of SPARK-19602, changes are made to support three part column name. In order to aid in the review and to reduce the diff, the test scenarios are separated out into this PR.
## How was this patch tested?
- This is a test only change. The individual test suites were run successfully.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/skambha/spark colResolutionTests
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17067.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17067
----
commit 9c2e24cc2fe7c194e3cde7c5ff623234a347774b
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-24T21:54:28Z
Add tests to cover scenarios for the column resolution
commit 778e3d6aa52b112c9126b9cc0492c49efac6ff4c
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-24T22:01:16Z
Add tests for global temp views
commit addb441c3579816d33cb996f073bb616679c6790
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-24T22:15:41Z
Add test with local view
commit 02cc8ed47989b84030e8f57fefd33a43b9cb4820
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-25T01:14:52Z
newline in end
commit 32d55e6d521047a9ba7b3fe0e7c75cd0a24cbab0
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-25T02:05:30Z
uppercase struct
commit 2f9937e5dc0fc02218c644083432f1bff241409f
Author: Sunitha Kambhampati <sk...@us.ibm.com>
Date: 2017-02-25T22:23:18Z
fix style and use ctas
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
Thanks! Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849606
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution.sql ---
@@ -0,0 +1,84 @@
+-- Tests covering different scenarios with qualified column names
+-- Scenario: column resolution scenarios with datasource table
+CREATE DATABASE mydb1;
+USE mydb1;
+CREATE TABLE t1 USING parquet AS SELECT 1 AS i1;
+
+CREATE DATABASE mydb2;
+USE mydb2;
+CREATE TABLE t1 USING parquet AS SELECT 20 AS i1;
+
+USE mydb1;
+SELECT i1 FROM t1;
+SELECT i1 FROM mydb1.t1;
+SELECT t1.i1 FROM t1;
+SELECT t1.i1 FROM mydb1.t1;
+
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM t1;
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM mydb1.t1;
+
+USE mydb2;
+SELECT i1 FROM t1;
+SELECT i1 FROM mydb1.t1;
+SELECT t1.i1 FROM t1;
+SELECT t1.i1 FROM mydb1.t1;
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM mydb1.t1;
+
+-- Scenario: resolve fully qualified table name in star expansion
+USE mydb1;
+SELECT t1.* FROM t1;
+SELECT mydb1.t1.* FROM mydb1.t1;
+SELECT t1.* FROM mydb1.t1;
+USE mydb2;
+SELECT t1.* FROM t1;
+-- TODO: Support this scenario
+SELECT mydb1.t1.* FROM mydb1.t1;
+SELECT t1.* FROM mydb1.t1;
+SELECT a.* FROM mydb1.t1 AS a;
+
+-- Scenario: resolve in case of subquery
+
+USE mydb1;
+CREATE TABLE t3 USING parquet AS SELECT * FROM VALUES (4,1), (3,1) AS t3(c1, c2);
+CREATE TABLE t4 USING parquet AS SELECT * FROM VALUES (4,1), (2,1) AS t4(c2, c3);
+
+SELECT * FROM t3 WHERE c1 IN (SELECT c2 FROM t4 WHERE t4.c3 = t3.c2);
+
+-- TODO: Support this scenario
+SELECT * FROM mydb1.t3 WHERE c1 IN
+ (SELECT mydb1.t4.c2 FROM mydb1.t4 WHERE mydb1.t4.c3 = mydb1.t3.c2);
+
+-- Scenario: column resolution scenarios in join queries
+SET spark.sql.crossJoin.enabled = true;
+
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM t1, mydb2.t1;
+
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM mydb1.t1, mydb2.t1;
+
+USE mydb2;
+-- TODO: Support this scenario
+SELECT mydb1.t1.i1 FROM t1, mydb1.t1;
+
+-- Scenario: Table with struct column
+USE mydb1;
+CREATE TABLE t5(i1 INT, t5 STRUCT<i1:INT, i2:INT>) USING parquet;
+INSERT INTO t5 VALUES(1, (2, 3));
+SELECT t5.i1 FROM t5;
+SELECT t5.t5.i1 FROM t5;
+SELECT t5.t5.i1 FROM mydb1.t5;
--- End diff --
Add two cases for verifying `*`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
Thanks much Xiao for the review and comments.
I have made the following changes:
- Separated out the -ve cases from the +ve cases.
- Moved positive tests and also the cases that should be supported into the SQLQueryTestSuite framework. A new test file columnresolution.sql and the corresponding master out file is added.
- Clean up the ColumnResolutionSuite to remove cases that are covered in the SQLQueryTestSuite
- I have kept the -ve cases in the ColumnResolutionSuite because the exprId shows up in the exception.
- I also wanted to cover a case against a hive serde table so I have kept those tests in the ColumnResolutionSuite
Please advise if we should move any others. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
Uh. Forgot one more point.
Could you move these test cases to our new end-to-end SQL query suite into `SQLQueryTestSuite` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/17067
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849099
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution-negative.sql ---
@@ -0,0 +1,36 @@
+-- Negative testcases for column resolution
+CREATE DATABASE mydb1;
+USE mydb1;
+CREATE TABLE t1 USING parquet AS SELECT 1 AS i1;
+
+CREATE DATABASE mydb2;
+USE mydb2;
+CREATE TABLE t1 USING parquet AS SELECT 20 AS i1;
+
+-- Negative tests: column resolution scenarios with ambiguous cases in join queries
+set spark.sql.crossJoin.enabled = true;
+USE mydb1;
+SELECT i1 FROM t1, mydb1.t1;
+SELECT t1.i1 FROM t1, mydb1.t1;
+SELECT mydb1.t1.i1 FROM t1, mydb1.t1;
+SELECT i1 FROM t1, mydb2.t1;
+SELECT t1.i1 FROM t1, mydb2.t1;
+USE mydb2;
+SELECT i1 FROM t1, mydb1.t1;
+SELECT t1.i1 FROM t1, mydb1.t1;
+SELECT i1 FROM t1, mydb2.t1;
+SELECT t1.i1 FROM t1, mydb2.t1;
+SELECT db1.t1.i1 FROM t1, mydb2.t1;
+
+-- Negative tests
+USE mydb1;
+SELECT mydb1.t1 FROM t1;
+SELECT t1.x.y.* FROM t1;
+SELECT t1 FROM mydb1.t1;
+USE mydb2;
+SELECT mydb1.t1.i1 FROM t1;
+
+-- reset
+set spark.sql.crossJoin.enabled = false;
+DROP DATABASE mydb1 CASCADE;
+DROP DATABASE mydb2 CASCADE;
--- End diff --
Please add one empty space after this line
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73722/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849191
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution-views.sql ---
@@ -0,0 +1,25 @@
+-- Tests for qualified column names for the view code-path
+-- Test scenario with Temporary view
+CREATE OR REPLACE TEMPORARY VIEW table1 AS SELECT 2 AS i1;
--- End diff --
`table1 ` -> `view1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378199
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+ test("column resolution scenarios with local temp view") {
+ val df = Seq(2).toDF("i1")
+ df.createOrReplaceTempView("table1")
+ withTempView("table1") {
+ checkAnswer(spark.sql("SELECT table1.* FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT * FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT table1.i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT a.i1 FROM table1 AS a"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1 AS a"), Row(2))
--- End diff --
How about these test cases for temporary views?
```
-- Test data.
CREATE OR REPLACE TEMPORARY VIEW testData AS SELECT * FROM VALUES
(1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2), (null, 1), (3, null), (null, null)
AS testData(a, b);
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73561/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73622/testReport)** for PR 17067 at commit [`1ae20ec`](https://github.com/apache/spark/commit/1ae20ec0a013358404f2b0d0e1ebadd60cdcd0b6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378740
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ColumnResolutionSuite.scala ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+class ColumnResolutionSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
--- End diff --
For the test cases you want to keep here, you can move it to `sql/core`. Why we need to test hive serde tables? Compared with data source tables, it is touching different code paths to resolve columns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73622/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73494 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73494/testReport)** for PR 17067 at commit [`2f9937e`](https://github.com/apache/spark/commit/2f9937e5dc0fc02218c644083432f1bff241409f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73561/testReport)** for PR 17067 at commit [`e4f347e`](https://github.com/apache/spark/commit/e4f347e648efba81c1be1ff679d7dd88967d408b).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
- Changes to the SQLQueryTestSuite framework to mask the exprId so I can add the -ve cases as well using this framework.
- Added -ve test cases to the SQLQueryTestSuite framework and so removed the hive specific test suite. For the hive table testcase, I will add that test as part of the actual code changes PR.
- I synced up the codeline and there was one test output inner-join.sql.out that needed a comment to be updated, so I have updated that as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849486
--- Diff: sql/core/src/test/resources/sql-tests/results/columnresolution-negative.sql.out ---
@@ -0,0 +1,240 @@
+-- Automatically generated by SQLQueryTestSuite
+-- Number of queries: 28
+
+
+-- !query 0
+CREATE DATABASE mydb1
+-- !query 0 schema
+struct<>
+-- !query 0 output
+
+
+
+-- !query 1
+USE mydb1
+-- !query 1 schema
+struct<>
+-- !query 1 output
+
+
+
+-- !query 2
+CREATE TABLE t1 USING parquet AS SELECT 1 AS i1
+-- !query 2 schema
+struct<>
+-- !query 2 output
+
+
+
+-- !query 3
+CREATE DATABASE mydb2
+-- !query 3 schema
+struct<>
+-- !query 3 output
+
+
+
+-- !query 4
+USE mydb2
+-- !query 4 schema
+struct<>
+-- !query 4 output
+
+
+
+-- !query 5
+CREATE TABLE t1 USING parquet AS SELECT 20 AS i1
+-- !query 5 schema
+struct<>
+-- !query 5 output
+
+
+
+-- !query 6
+set spark.sql.crossJoin.enabled = true
+-- !query 6 schema
+struct<key:string,value:string>
+-- !query 6 output
+spark.sql.crossJoin.enabled true
+
+
+-- !query 7
+USE mydb1
+-- !query 7 schema
+struct<>
+-- !query 7 output
+
+
+
+-- !query 8
+SELECT i1 FROM t1, mydb1.t1
+-- !query 8 schema
+struct<>
+-- !query 8 output
+org.apache.spark.sql.AnalysisException
+Reference 'i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7
+
+
+-- !query 9
+SELECT t1.i1 FROM t1, mydb1.t1
+-- !query 9 schema
+struct<>
+-- !query 9 output
+org.apache.spark.sql.AnalysisException
+Reference 't1.i1' is ambiguous, could be: i1#x, i1#x.; line 1 pos 7
--- End diff --
To the other reviewer, this is the reason why we need to make a change in `SQLQueryTestSuite.scala`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73479 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73479/testReport)** for PR 17067 at commit [`2f9937e`](https://github.com/apache/spark/commit/2f9937e5dc0fc02218c644083432f1bff241409f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73779 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73779/testReport)** for PR 17067 at commit [`b2e411c`](https://github.com/apache/spark/commit/b2e411c859f4266428db6b4fbf55d89a101e50bf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73722 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73722/testReport)** for PR 17067 at commit [`5594eb0`](https://github.com/apache/spark/commit/5594eb0864376bbac617bf744755330f1e7bff49).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849138
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution-negative.sql ---
@@ -0,0 +1,36 @@
+-- Negative testcases for column resolution
+CREATE DATABASE mydb1;
+USE mydb1;
+CREATE TABLE t1 USING parquet AS SELECT 1 AS i1;
+
+CREATE DATABASE mydb2;
+USE mydb2;
+CREATE TABLE t1 USING parquet AS SELECT 20 AS i1;
+
+-- Negative tests: column resolution scenarios with ambiguous cases in join queries
+set spark.sql.crossJoin.enabled = true;
+USE mydb1;
+SELECT i1 FROM t1, mydb1.t1;
+SELECT t1.i1 FROM t1, mydb1.t1;
+SELECT mydb1.t1.i1 FROM t1, mydb1.t1;
+SELECT i1 FROM t1, mydb2.t1;
+SELECT t1.i1 FROM t1, mydb2.t1;
+USE mydb2;
+SELECT i1 FROM t1, mydb1.t1;
+SELECT t1.i1 FROM t1, mydb1.t1;
+SELECT i1 FROM t1, mydb2.t1;
+SELECT t1.i1 FROM t1, mydb2.t1;
+SELECT db1.t1.i1 FROM t1, mydb2.t1;
+
+-- Negative tests
+USE mydb1;
+SELECT mydb1.t1 FROM t1;
+SELECT t1.x.y.* FROM t1;
+SELECT t1 FROM mydb1.t1;
+USE mydb2;
+SELECT mydb1.t1.i1 FROM t1;
+
+-- reset
+set spark.sql.crossJoin.enabled = false;
--- End diff --
Move this to line 24
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73494 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73494/testReport)** for PR 17067 at commit [`2f9937e`](https://github.com/apache/spark/commit/2f9937e5dc0fc02218c644083432f1bff241409f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
Generally, it looks good to me.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103379909
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/ColumnResolutionSuite.scala ---
@@ -0,0 +1,173 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, QueryTest, Row}
+import org.apache.spark.sql.hive.test.TestHiveSingleton
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SQLTestUtils
+
+class ColumnResolutionSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
--- End diff --
The logic to resolve the column in the LogicalPlan is same - there is no change there. I wanted to test the hive table to make sure that the qualifier information is correctly set. We update the qualifier info in MetastoreRelation so wanted to have coverage for hive table.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378392
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+ test("column resolution scenarios with local temp view") {
+ val df = Seq(2).toDF("i1")
+ df.createOrReplaceTempView("table1")
+ withTempView("table1") {
+ checkAnswer(spark.sql("SELECT table1.* FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT * FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT table1.i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT a.i1 FROM table1 AS a"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1 AS a"), Row(2))
--- End diff --
Sure, let me look at converting these too. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
Thank you for working on this! I like the coverage of qualified column names.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73622/testReport)** for PR 17067 at commit [`1ae20ec`](https://github.com/apache/spark/commit/1ae20ec0a013358404f2b0d0e1ebadd60cdcd0b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73779/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378807
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution.sql ---
@@ -0,0 +1,82 @@
+-- Scenario: column resolution scenarios with datasource table
+CREATE DATABASE mydb1;
+use mydb1;
--- End diff --
Please use upper case for SQL keywords.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103378842
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+ test("column resolution scenarios with local temp view") {
+ val df = Seq(2).toDF("i1")
+ df.createOrReplaceTempView("table1")
+ withTempView("table1") {
+ checkAnswer(spark.sql("SELECT table1.* FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT * FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT table1.i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT a.i1 FROM table1 AS a"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1 AS a"), Row(2))
--- End diff --
Also doable for global temporary view, I think
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73561/testReport)** for PR 17067 at commit [`e4f347e`](https://github.com/apache/spark/commit/e4f347e648efba81c1be1ff679d7dd88967d408b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73722/testReport)** for PR 17067 at commit [`5594eb0`](https://github.com/apache/spark/commit/5594eb0864376bbac617bf744755330f1e7bff49).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849396
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala ---
@@ -228,12 +228,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSQLContext {
if (isSorted(df.queryExecution.analyzed)) (schema, answer) else (schema, answer.sorted)
} catch {
- case a: AnalysisException if a.plan.nonEmpty =>
+ case a: AnalysisException =>
// Do not output the logical plan tree which contains expression IDs.
// Also implement a crude way of masking expression IDs in the error message
// with a generic pattern "###".
+ val msg = if (a.plan.nonEmpty) a.getSimpleMessage else a.getMessage
(StructType(Seq.empty),
- Seq(a.getClass.getName, a.getSimpleMessage.replaceAll("#\\d+", "#x")))
+ Seq(a.getClass.getName, msg.replaceAll("#\\d+", "#x")))
--- End diff --
Nit: `(StructType(Seq.empty), Seq(a.getClass.getName, msg.replaceAll("#\\d+", "#x")))`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103849231
--- Diff: sql/core/src/test/resources/sql-tests/inputs/columnresolution-views.sql ---
@@ -0,0 +1,25 @@
+-- Tests for qualified column names for the view code-path
+-- Test scenario with Temporary view
+CREATE OR REPLACE TEMPORARY VIEW table1 AS SELECT 2 AS i1;
+SELECT table1.* FROM table1;
+SELECT * FROM table1;
+SELECT table1.i1 FROM table1;
+SELECT i1 FROM table1;
+SELECT a.i1 FROM table1 AS a;
+SELECT i1 FROM table1 AS a;
+-- cleanup
+DROP VIEW table1;
+
+-- Test scenario with Global Temp view
+CREATE OR REPLACE GLOBAL TEMPORARY VIEW t1 as SELECT 1 as i1;
--- End diff --
`t1` -> `view1`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
Some test cases are negative; some test cases are expected to be supported. Could you split them from the positive test cases? Now some test cases are pretty large.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on a diff in the pull request:
https://github.com/apache/spark/pull/17067#discussion_r103756083
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -52,6 +52,19 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
+ test("column resolution scenarios with local temp view") {
+ val df = Seq(2).toDF("i1")
+ df.createOrReplaceTempView("table1")
+ withTempView("table1") {
+ checkAnswer(spark.sql("SELECT table1.* FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT * FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT table1.i1 FROM table1"), Row(2))
+ checkAnswer(spark.sql("SELECT a.i1 FROM table1 AS a"), Row(2))
+ checkAnswer(spark.sql("SELECT i1 FROM table1 AS a"), Row(2))
--- End diff --
Hi Xiao, I have moved my new local temp view tests and the global temp view tests to the SQLQueryTestSuite framework as well. Please take a look. Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73494/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/73479/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/17067
**[Test build #73779 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/73779/testReport)** for PR 17067 at commit [`b2e411c`](https://github.com/apache/spark/commit/b2e411c859f4266428db6b4fbf55d89a101e50bf).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17067
We can move the remaining test cases to `SQLQueryTestSuite `, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/17067
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #17067: [SPARK-19602][SQL][TESTS] Add tests for qualified column...
Posted by skambha <gi...@git.apache.org>.
Github user skambha commented on the issue:
https://github.com/apache/spark/pull/17067
Thanks a lot Xiao.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org