You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jiangxb1987 <gi...@git.apache.org> on 2017/01/22 12:13:09 UTC
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/16674
[SPARK-19331][SQL][TESTS] Improve the test coverage of SQLViewSuite
## What changes were proposed in this pull request?
Improve the test coverage of SQLViewSuite, cover the following cases:
1. view resolution(possibly a referenced table/view have changed after the view creation);
2. handle a view with user specified column names;
3. improve the test cases for a nested view.
Also added a test case for cyclic view reference, which is a known issue that is not fixed yet.
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark view-test
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16674.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16674
----
commit 60d927da2395784687e918ae17f7c90d89c7d19e
Author: jiangxingbo <ji...@gmail.com>
Date: 2017-01-22T11:48:06Z
update SQLViewSuite
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72573 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72573/testReport)** for PR 16674 at commit [`2e201c2`](https://github.com/apache/spark/commit/2e201c26415104d4b9d417399cb87db0c8d07308).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72914/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100906893
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -15,29 +15,34 @@
* limitations under the License.
*/
-package org.apache.spark.sql.hive.execution
+package org.apache.spark.sql.execution
-import org.apache.spark.sql.{AnalysisException, QueryTest, Row, SaveMode}
+import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
-import org.apache.spark.sql.hive.test.TestHiveSingleton
-import org.apache.spark.sql.test.SQLTestUtils
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.test.{SharedSQLContext, SQLTestUtils}
+
+class SimpleSQLViewSuite extends SQLViewSuite with SharedSQLContext {
+}
/**
* A suite for testing view related functionality.
*/
-class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
- import spark.implicits._
+abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
--- End diff --
how about we make a `trait SparkSessionProvider` and make `SharedSparkContext` and `TestHiveSingleton` both extend it? then we can make `SQLViewSuite` extends `SparkSessionProvider` and fix this problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72742 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72742/testReport)** for PR 16674 at commit [`e8fc8c7`](https://github.com/apache/spark/commit/e8fc8c775315cdecb92712ded540a638191f07c2).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72853/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72853/testReport)** for PR 16674 at commit [`4872918`](https://github.com/apache/spark/commit/48729186467e9b9d19ef86e1b635a9746735bfa0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72742/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16674
@gatorsmile @cloud-fan Could you please look at this when you have time? Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/16674
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002440
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
--- End diff --
This test case is rewritten to `test("correctly parse a nested view")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r101111632
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,18 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val db = u.tableIdentifier.database.orElse(defaultDatabase)
+ val tableIdentWithDb = u.tableIdentifier.copy(database = db)
try {
- val tableIdentWithDb = u.tableIdentifier.copy(
- database = u.tableIdentifier.database.orElse(defaultDatabase))
catalog.lookupRelation(tableIdentWithDb, u.alias)
} catch {
case _: NoSuchTableException =>
- u.failAnalysis(s"Table or view not found: ${u.tableName}")
+ u.failAnalysis(s"Table or view not found: ${tableIdentWithDb.unquotedString}")
+ // If the database is defined and that database is not found, throw an AnalysisException.
+ // Note that if the database is not defined, it is possible we are looking up a temp view.
+ case _: NoSuchDatabaseException if db.isDefined =>
--- End diff --
let's remove this if, since we can't get the current database in `Analyzer`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72742 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72742/testReport)** for PR 16674 at commit [`e8fc8c7`](https://github.com/apache/spark/commit/e8fc8c775315cdecb92712ded540a638191f07c2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002779
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
- withTable("testTable") {
- spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
+ withTable("tab1") {
--- End diff --
The name `testTable` has been existed, so we change the table name to `tab1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100684983
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +534,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
--- End diff --
The following test cases are moved to `HiveSQLViewSuite `
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #71806 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71806/testReport)** for PR 16674 at commit [`60d927d`](https://github.com/apache/spark/commit/60d927da2395784687e918ae17f7c90d89c7d19e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #71806 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/71806/testReport)** for PR 16674 at commit [`60d927d`](https://github.com/apache/spark/commit/60d927da2395784687e918ae17f7c90d89c7d19e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72573/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100595532
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
--- End diff --
actually you moved it [here](https://github.com/apache/spark/pull/16674/files#diff-e09027394aebb11c880b375693f59fafR495)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r97231085
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLViewSuite.scala ---
@@ -739,18 +706,7 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
withTable("testTable") {
spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
withView("testView") {
- val testView = CatalogTable(
- identifier = TableIdentifier("testView"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewOriginalText = Some("SELECT * FROM testTable"),
- viewText = Some("SELECT * FROM testTable"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(testView, ignoreIfExists = false)
+ sql("CREATE VIEW testView AS SELECT * FROM testTable")
--- End diff --
What is the reason?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100689894
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,17 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val tableIdentWithDb = u.tableIdentifier.copy(
+ database = u.tableIdentifier.database.orElse(defaultDatabase))
--- End diff --
For a temp view, we should not set the database of the tableIdentifier. Perhaps we can change the case statement to:
```
case _: NoSuchDatabaseException if db.isDefined =>
u.failAnalysis(s"Table or view not found: ${tableIdentWithDb.unquotedString}, the " +
s"database ${db.get} doesn't exsits.")
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16674
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100671808
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
--- End diff --
The problem is that `SharedSQLContext` also overrides `beforeAll()` and `afterAll()`, I haven't figure out a good way to resolve the issue, do you have any suggestions?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72567/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100685083
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
--- End diff --
Yes, we can remove both by doing something like
```Scala
abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
import testImplicits._
protected override def beforeAll(): Unit = {
super.beforeAll()
// Create a simple table with two columns: id and id1
spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
}
protected override def afterAll(): Unit = {
try {
spark.sql(s"DROP TABLE IF EXISTS jt")
} finally {
super.afterAll()
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/71806/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72677 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72677/testReport)** for PR 16674 at commit [`664b53a`](https://github.com/apache/spark/commit/664b53a86ceecf2e15808f01abde9ad88fd7c4fe).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16674
Happy New Year! : )
I double checked all the test case change. The changes makes sense to me. Could you resolve the conflicts?
BTW, a general suggestion. Could you please add the comments when you move the codes in the PR? It can help others review the codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100238713
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS jt")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ import testImplicits._
+
+ test("create a permanent/temp view using a hive, built-in, and permanent user function") {
+ val permanentFuncName = "myUpper"
+ val permanentFuncClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ val builtInFuncNameInLowerCase = "abs"
+ val builtInFuncNameInMixedCase = "aBs"
+ val hiveFuncName = "histogram_numeric"
+
+ withUserDefinedFunction(permanentFuncName -> false) {
+ sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
+ withTable("tab1") {
+ (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
+ Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
+ withView("view1") {
+ sql(
+ s"""
+ |CREATE $viewMode view1
+ |AS SELECT
+ |$permanentFuncName(str),
+ |$builtInFuncNameInLowerCase(id),
+ |$builtInFuncNameInMixedCase(id) as aBs,
+ |$hiveFuncName(id, 5) over()
+ |FROM tab1
+ """.stripMargin)
+ checkAnswer(sql("select count(*) FROM view1"), Row(10))
+ }
+ }
+ }
+ }
+ }
+
+ test("create a permanent/temp view using a temporary function") {
+ val tempFunctionName = "temp"
+ val functionClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ withUserDefinedFunction(tempFunctionName -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
+ withView("view1", "tempView1") {
+ withTable("tab1") {
+ (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
+
+ // temporary view
+ sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
+ checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
+
+ // permanent view
+ val e = intercept[AnalysisException] {
+ sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
+ }.getMessage
+ assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
+ s"a temporary function `$tempFunctionName`"))
+ }
+ }
+ }
+ }
+
+ test("create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to 9).map(i => Row(i)))
+ }
+ }
+
+ test("create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the new flag fix it.
+ withTable("parTable") {
+ withView("testView") {
+ val df = Seq(1 -> "a").toDF("i", "j")
+ df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+ sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+ checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
+ }
+ }
+
+ test("create hive view for joined tables") {
--- End diff --
I moved this to `HiveSQLViewSuite` just because the test name describes itself to be hive specific.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72914 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72914/testReport)** for PR 16674 at commit [`aa0cf0a`](https://github.com/apache/spark/commit/aa0cf0a80dfb00b7d367b2a58ecf0ae6ddeee16d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class NoSuchDatabaseException(val db: String) extends AnalysisException(s\"Database '$db' not found\")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72774/testReport)** for PR 16674 at commit [`2f00404`](https://github.com/apache/spark/commit/2f00404bf12545a67791f22ac8ebd19b91564c69).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16674
A late LGTM : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100596259
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
--- End diff --
can we put this in `SQLViewSuite`? then we don't need to duplicate this logic in `SimpleSQLViewSuite` and `HiveSQLViewSuite`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002542
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
--- End diff --
This test case is rewritten to `test("correctly resolve a view in a self join")`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16674
I saw the PR deleted a few test cases. Could you post the reasons you did?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100235360
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS jt")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ import testImplicits._
+
+ test("create a permanent/temp view using a hive, built-in, and permanent user function") {
+ val permanentFuncName = "myUpper"
+ val permanentFuncClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ val builtInFuncNameInLowerCase = "abs"
+ val builtInFuncNameInMixedCase = "aBs"
+ val hiveFuncName = "histogram_numeric"
+
+ withUserDefinedFunction(permanentFuncName -> false) {
+ sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
+ withTable("tab1") {
+ (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
+ Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
+ withView("view1") {
+ sql(
+ s"""
+ |CREATE $viewMode view1
+ |AS SELECT
+ |$permanentFuncName(str),
+ |$builtInFuncNameInLowerCase(id),
+ |$builtInFuncNameInMixedCase(id) as aBs,
+ |$hiveFuncName(id, 5) over()
+ |FROM tab1
+ """.stripMargin)
+ checkAnswer(sql("select count(*) FROM view1"), Row(10))
+ }
+ }
+ }
+ }
+ }
+
+ test("create a permanent/temp view using a temporary function") {
+ val tempFunctionName = "temp"
+ val functionClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ withUserDefinedFunction(tempFunctionName -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
+ withView("view1", "tempView1") {
+ withTable("tab1") {
+ (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
+
+ // temporary view
+ sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
+ checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
+
+ // permanent view
+ val e = intercept[AnalysisException] {
+ sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
+ }.getMessage
+ assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
+ s"a temporary function `$tempFunctionName`"))
+ }
+ }
+ }
+ }
+
+ test("create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to 9).map(i => Row(i)))
+ }
+ }
+
+ test("create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the new flag fix it.
+ withTable("parTable") {
+ withView("testView") {
+ val df = Seq(1 -> "a").toDF("i", "j")
+ df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+ sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+ checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
+ }
+ }
+
+ test("create hive view for joined tables") {
--- End diff --
Any reason why this is Hive only?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100689818
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
--- End diff --
The call of `SharedSQLContext.afterAll()` is before that of `SQLViewSuite.afterAll()`, so the variable `spark` has been set to null in `SQLViewSuite.afterAll()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100678170
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +542,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
--- End diff --
can we merge them? Seems these 2 tests do have some common part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100672477
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -398,26 +472,6 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for json table") {
--- End diff --
The comment of @jiangxb1987 is out of dated. After leaving this comment, he moved the test cases to `SQLViewSuite` based on my comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100678107
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,17 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val tableIdentWithDb = u.tableIdentifier.copy(
+ database = u.tableIdentifier.database.orElse(defaultDatabase))
--- End diff --
`u.tableIdentifier.database.orElse(defaultDatabase).orElse(Some(conf.defaultDB))`, or we can't guarantee we have the database part
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002125
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -398,26 +472,6 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for json table") {
--- End diff --
These test cases have been moved to `HiveSQLViewSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002487
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
--- End diff --
This test case is duplicated with `test("CTE within view")`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72677 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72677/testReport)** for PR 16674 at commit [`664b53a`](https://github.com/apache/spark/commit/664b53a86ceecf2e15808f01abde9ad88fd7c4fe).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002647
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
--- End diff --
This test case has been moved to `HiveSQLViewSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100397916
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS jt")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ import testImplicits._
+
+ test("create a permanent/temp view using a hive, built-in, and permanent user function") {
+ val permanentFuncName = "myUpper"
+ val permanentFuncClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ val builtInFuncNameInLowerCase = "abs"
+ val builtInFuncNameInMixedCase = "aBs"
+ val hiveFuncName = "histogram_numeric"
+
+ withUserDefinedFunction(permanentFuncName -> false) {
+ sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
+ withTable("tab1") {
+ (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
+ Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
+ withView("view1") {
+ sql(
+ s"""
+ |CREATE $viewMode view1
+ |AS SELECT
+ |$permanentFuncName(str),
+ |$builtInFuncNameInLowerCase(id),
+ |$builtInFuncNameInMixedCase(id) as aBs,
+ |$hiveFuncName(id, 5) over()
+ |FROM tab1
+ """.stripMargin)
+ checkAnswer(sql("select count(*) FROM view1"), Row(10))
+ }
+ }
+ }
+ }
+ }
+
+ test("create a permanent/temp view using a temporary function") {
+ val tempFunctionName = "temp"
+ val functionClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ withUserDefinedFunction(tempFunctionName -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
+ withView("view1", "tempView1") {
+ withTable("tab1") {
+ (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
+
+ // temporary view
+ sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
+ checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
+
+ // permanent view
+ val e = intercept[AnalysisException] {
+ sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
+ }.getMessage
+ assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
+ s"a temporary function `$tempFunctionName`"))
+ }
+ }
+ }
+ }
+
+ test("create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to 9).map(i => Row(i)))
+ }
+ }
+
+ test("create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the new flag fix it.
+ withTable("parTable") {
+ withView("testView") {
+ val df = Seq(1 -> "a").toDF("i", "j")
+ df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+ sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+ checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
+ }
+ }
+
+ test("create hive view for joined tables") {
--- End diff --
: ) Could you ignore the test case names and make `HiveSQLViewSuite` as small as possible?
If we move them to `SQLViewSuite`, they will also be executed in `HiveSQLViewSuite`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100670864
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +542,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
--- End diff --
I'm afraid the former test case is on `CREATE VIEW`, and this test case is on view resolution, so they are not duplicated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100596077
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +542,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
- withTable("testTable") {
- spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
+ withTable("tab1") {
+ spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("tab1")
withView("testView") {
- val testView = CatalogTable(
- identifier = TableIdentifier("testView"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM testTable"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(testView, ignoreIfExists = false)
+ sql("CREATE VIEW testView(x, y) AS SELECT * FROM tab1")
// Correctly resolve a view with custom column names.
checkAnswer(sql("SELECT * FROM testView ORDER BY x"), (1 to 9).map(i => Row(i, i + 1)))
// Correctly resolve a view when the referenced table schema changes.
spark.range(1, 10).selectExpr("id", "id + id dummy", "id + 1 id1")
- .write.mode(SaveMode.Overwrite).saveAsTable("testTable")
+ .write.mode(SaveMode.Overwrite).saveAsTable("tab1")
checkAnswer(sql("SELECT * FROM testView ORDER BY x"), (1 to 9).map(i => Row(i, i + 1)))
// Throw an AnalysisException if the column name is not found.
spark.range(1, 10).selectExpr("id", "id + 1 dummy")
- .write.mode(SaveMode.Overwrite).saveAsTable("testTable")
+ .write.mode(SaveMode.Overwrite).saveAsTable("tab1")
intercept[AnalysisException](sql("SELECT * FROM testView"))
}
}
}
test("resolve a view when the dataTypes of referenced table columns changed") {
- withTable("testTable") {
- spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
+ withTable("tab1") {
--- End diff --
can we use `jt` instead?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100595967
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +542,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
--- End diff --
is this test duplicated with https://github.com/apache/spark/pull/16674/files#diff-e09027394aebb11c880b375693f59fafR273 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/16674
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r97231186
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/SQLViewSuite.scala ---
@@ -739,18 +706,7 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
withTable("testTable") {
spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
withView("testView") {
- val testView = CatalogTable(
- identifier = TableIdentifier("testView"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewOriginalText = Some("SELECT * FROM testTable"),
- viewText = Some("SELECT * FROM testTable"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(testView, ignoreIfExists = false)
+ sql("CREATE VIEW testView AS SELECT * FROM testTable")
--- End diff --
How about creating a new test case? First, create a view using a regular SQL statement, and then fetch the metadata and verify the fetched metadata is still matching what we expect?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72774/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100684516
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,17 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val tableIdentWithDb = u.tableIdentifier.copy(
+ database = u.tableIdentifier.database.orElse(defaultDatabase))
--- End diff --
```Scala
val db = u.tableIdentifier.database.orElse(defaultDatabase).orElse(Some(conf.defaultDB))
```
Then we can use it in the error message.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72774/testReport)** for PR 16674 at commit [`2f00404`](https://github.com/apache/spark/commit/2f00404bf12545a67791f22ac8ebd19b91564c69).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100593400
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -624,6 +624,9 @@ class Analyzer(
} catch {
case _: NoSuchTableException =>
u.failAnalysis(s"Table or view not found: ${u.tableName}")
+ // If the database is not valid, also throw an AnalysisException.
+ case _: NoSuchDatabaseException =>
+ u.failAnalysis(s"Table or view not found: ${u.tableName}")
--- End diff --
shall we use the qualified name? or users are hard to find out that the database doesn't exist, given only the table name.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r101352242
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,18 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val tableIdentWithDb = u.tableIdentifier.copy(database = u.tableIdentifier.database
--- End diff --
nvm, I fixed it while merging
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/72677/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on the issue:
https://github.com/apache/spark/pull/16674
@gatorsmile The only test cast that I removed is `test("correctly resolve a view with CTE")`, which is duplicate with the existing `test("CTE within view")`.
For testing without using Hive metastore, I'll update the test suites follow your suggestion, maybe later in a few days, since you know, the Chinese New Year is coming :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72853/testReport)** for PR 16674 at commit [`4872918`](https://github.com/apache/spark/commit/48729186467e9b9d19ef86e1b635a9746735bfa0).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72914 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72914/testReport)** for PR 16674 at commit [`aa0cf0a`](https://github.com/apache/spark/commit/aa0cf0a80dfb00b7d367b2a58ecf0ae6ddeee16d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100678180
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
--- End diff --
what do you mean? In `SQLViewSuite.beforeAll`, we can still call `super.beforeAll`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100670964
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +542,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
- // make sure the new flag can handle some complex cases like join and schema change.
- withTable("jt1", "jt2") {
- spark.range(1, 10).toDF("id1").write.format("json").saveAsTable("jt1")
- spark.range(1, 10).toDF("id2").write.format("json").saveAsTable("jt2")
- sql("CREATE VIEW testView AS SELECT * FROM jt1 JOIN jt2 ON id1 == id2")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- val df = (1 until 10).map(i => i -> i).toDF("id1", "newCol")
- df.write.format("json").mode(SaveMode.Overwrite).saveAsTable("jt1")
- checkAnswer(sql("SELECT * FROM testView ORDER BY id1"), (1 to 9).map(i => Row(i, i)))
-
- sql("DROP VIEW testView")
- }
- }
-
- test("SPARK-14933 - create view from hive parquet table") {
- withTable("t_part") {
- withView("v_part") {
- spark.sql("create table t_part stored as parquet as select 1 as a, 2 as b")
- spark.sql("create view v_part as select * from t_part")
- checkAnswer(
- sql("select * from t_part"),
- sql("select * from v_part"))
- }
- }
- }
-
- test("SPARK-14933 - create view from hive orc table") {
- withTable("t_orc") {
- withView("v_orc") {
- spark.sql("create table t_orc stored as orc as select 1 as a, 2 as b")
- spark.sql("create view v_orc as select * from t_orc")
- checkAnswer(
- sql("select * from t_orc"),
- sql("select * from v_orc"))
- }
- }
- }
-
- test("create a permanent/temp view using a hive, built-in, and permanent user function") {
- val permanentFuncName = "myUpper"
- val permanentFuncClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- val builtInFuncNameInLowerCase = "abs"
- val builtInFuncNameInMixedCase = "aBs"
- val hiveFuncName = "histogram_numeric"
-
- withUserDefinedFunction(permanentFuncName -> false) {
- sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
- withTable("tab1") {
- (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
- Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
- withView("view1") {
- sql(
- s"""
- |CREATE $viewMode view1
- |AS SELECT
- |$permanentFuncName(str),
- |$builtInFuncNameInLowerCase(id),
- |$builtInFuncNameInMixedCase(id) as aBs,
- |$hiveFuncName(id, 5) over()
- |FROM tab1
- """.stripMargin)
- checkAnswer(sql("select count(*) FROM view1"), Row(10))
- }
- }
- }
- }
- }
-
- test("create a permanent/temp view using a temporary function") {
- val tempFunctionName = "temp"
- val functionClass =
- classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
- withUserDefinedFunction(tempFunctionName -> true) {
- sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
- withView("view1", "tempView1") {
- withTable("tab1") {
- (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
-
- // temporary view
- sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
- checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
-
- // permanent view
- val e = intercept[AnalysisException] {
- sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
- }.getMessage
- assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
- s"a temporary function `$tempFunctionName`"))
- }
- }
- }
- }
-
- test("correctly resolve a nested view") {
- withTempDatabase { db =>
- withView(s"$db.view1", s"$db.view2") {
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2", Some(db)),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view1"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> db,
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "x",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "y"))
- activateDatabase(db) {
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
- }
- }
- }
- }
-
- test("correctly resolve a view with CTE") {
- withView("cte_view") {
- val cte_view = CatalogTable(
- identifier = TableIdentifier("cte_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("n", "int"),
- viewText = Some("WITH w AS (SELECT 1 AS n) SELECT n FROM w"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "1",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "n"))
- hiveContext.sessionState.catalog.createTable(cte_view, ignoreIfExists = false)
- checkAnswer(sql("SELECT * FROM cte_view"), Row(1))
- }
- }
-
- test("correctly resolve a view in a self join") {
- withView("join_view") {
- val join_view = CatalogTable(
- identifier = TableIdentifier("join_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(join_view, ignoreIfExists = false)
- checkAnswer(
- sql("SELECT * FROM join_view t1 JOIN join_view t2 ON t1.id = t2.id ORDER BY t1.id"),
- (1 to 9).map(i => Row(i, i, i, i)))
- }
- }
-
- private def assertInvalidReference(query: String): Unit = {
- val e = intercept[AnalysisException] {
- sql(query)
- }.getMessage
- assert(e.contains("Table or view not found"))
- }
-
test("error handling: fail if the referenced table or view is invalid") {
withView("view1", "view2", "view3") {
// Fail if the referenced table is defined in a invalid database.
- val view1 = CatalogTable(
- identifier = TableIdentifier("view1"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_db.jt"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view1, ignoreIfExists = false)
+ withTempDatabase { db =>
+ withTable(s"$db.table1") {
+ activateDatabase(db) {
+ sql("CREATE TABLE table1(a int, b string) USING parquet")
+ sql("CREATE VIEW default.view1 AS SELECT * FROM table1")
+ }
+ }
+ }
assertInvalidReference("SELECT * FROM view1")
// Fail if the referenced table is invalid.
- val view2 = CatalogTable(
- identifier = TableIdentifier("view2"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM invalid_table"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view2, ignoreIfExists = false)
+ withTable("table2") {
+ sql("CREATE TABLE table2(a int, b string) USING parquet")
+ sql("CREATE VIEW view2 AS SELECT * FROM table2")
+ }
assertInvalidReference("SELECT * FROM view2")
// Fail if the referenced view is invalid.
- val view3 = CatalogTable(
- identifier = TableIdentifier("view3"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("id", "long").add("id1", "long"),
- viewText = Some("SELECT * FROM view2"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(view3, ignoreIfExists = false)
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ sql("CREATE VIEW view3 AS SELECT * FROM testView")
+ }
assertInvalidReference("SELECT * FROM view3")
}
}
- test("make sure we can resolve view created by old version of Spark") {
- withTable("hive_table") {
- withView("old_view") {
- spark.sql("CREATE TABLE hive_table AS SELECT 1 AS a, 2 AS b")
- // The views defined by older versions of Spark(before 2.2) will have empty view default
- // database name, and all the relations referenced in the viewText will have database part
- // defined.
- val view = CatalogTable(
- identifier = TableIdentifier("old_view"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("a", "int").add("b", "int"),
- viewText = Some("SELECT `gen_attr_0` AS `a`, `gen_attr_1` AS `b` FROM (SELECT " +
- "`gen_attr_0`, `gen_attr_1` FROM (SELECT `a` AS `gen_attr_0`, `b` AS " +
- "`gen_attr_1` FROM hive_table) AS gen_subquery_0) AS hive_table")
- )
- hiveContext.sessionState.catalog.createTable(view, ignoreIfExists = false)
- val df = sql("SELECT * FROM old_view")
- // Check the output rows.
- checkAnswer(df, Row(1, 2))
- // Check the output schema.
- assert(df.schema.sameType(view.schema))
- }
+ test("correctly resolve a view in a self join") {
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT * FROM jt")
+ checkAnswer(
+ sql("SELECT * FROM testView t1 JOIN testView t2 ON t1.id = t2.id ORDER BY t1.id"),
+ (1 to 9).map(i => Row(i, i, i, i)))
}
}
test("resolve a view with custom column names") {
- withTable("testTable") {
- spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
+ withTable("tab1") {
+ spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("tab1")
withView("testView") {
- val testView = CatalogTable(
- identifier = TableIdentifier("testView"),
- tableType = CatalogTableType.VIEW,
- storage = CatalogStorageFormat.empty,
- schema = new StructType().add("x", "long").add("y", "long"),
- viewText = Some("SELECT * FROM testTable"),
- properties = Map(CatalogTable.VIEW_DEFAULT_DATABASE -> "default",
- CatalogTable.VIEW_QUERY_OUTPUT_NUM_COLUMNS -> "2",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}0" -> "id",
- s"${CatalogTable.VIEW_QUERY_OUTPUT_COLUMN_NAME_PREFIX}1" -> "id1"))
- hiveContext.sessionState.catalog.createTable(testView, ignoreIfExists = false)
+ sql("CREATE VIEW testView(x, y) AS SELECT * FROM tab1")
// Correctly resolve a view with custom column names.
checkAnswer(sql("SELECT * FROM testView ORDER BY x"), (1 to 9).map(i => Row(i, i + 1)))
// Correctly resolve a view when the referenced table schema changes.
spark.range(1, 10).selectExpr("id", "id + id dummy", "id + 1 id1")
- .write.mode(SaveMode.Overwrite).saveAsTable("testTable")
+ .write.mode(SaveMode.Overwrite).saveAsTable("tab1")
checkAnswer(sql("SELECT * FROM testView ORDER BY x"), (1 to 9).map(i => Row(i, i + 1)))
// Throw an AnalysisException if the column name is not found.
spark.range(1, 10).selectExpr("id", "id + 1 dummy")
- .write.mode(SaveMode.Overwrite).saveAsTable("testTable")
+ .write.mode(SaveMode.Overwrite).saveAsTable("tab1")
intercept[AnalysisException](sql("SELECT * FROM testView"))
}
}
}
test("resolve a view when the dataTypes of referenced table columns changed") {
- withTable("testTable") {
- spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("testTable")
+ withTable("tab1") {
--- End diff --
For this test case I think we'd better create a new table, because we would overwrite the table in the test case, if we use `jt` then we have to recover the table content and that may be error prone.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/16674
I think we are having a big gap for testing view supports without using Hive metastore. So far, we only have a single test case you added. My proposal here is to move the whole test suite to the SQL module from the Hive module. Then, create another test suite in Hive that extends the suite in SQL.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002271
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -452,311 +506,96 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for joined tables") {
--- End diff --
These test cases have been moved to `HiveSQLViewSuite`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100594350
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -233,6 +252,42 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
+ test("correctly parse a nested view") {
+ withTempDatabase { db =>
+ withView("view1", "view2", s"$db.view3") {
+ sql("CREATE VIEW view1(x, y) AS SELECT * FROM jt")
+
+ // Create a nested view in the same database.
+ sql("CREATE VIEW view2(id, id1) AS SELECT * FROM view1")
+ checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
+
+ // Create a nested view in a different database.
+ activateDatabase(db) {
+ sql(s"CREATE VIEW $db.view3(id, id1) AS SELECT * FROM default.view1")
+ checkAnswer(sql("SELECT * FROM view3 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
+ }
+ }
+ }
+ }
+
+ test("correctly parse a view with custom column names") {
+ withTable("tab1") {
+ spark.range(1, 10).selectExpr("id", "id + 1 id1").write.saveAsTable("tab1")
+ withView("testView1", "testView2") {
+ // Correctly create a view with custom column names
+ sql("CREATE VIEW testView1(x, y) AS SELECT * FROM tab1")
--- End diff --
can we just use `jt` and avoid creating `tab1`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100672397
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -624,6 +624,9 @@ class Analyzer(
} catch {
case _: NoSuchTableException =>
u.failAnalysis(s"Table or view not found: ${u.tableName}")
+ // If the database is not valid, also throw an AnalysisException.
+ case _: NoSuchDatabaseException =>
+ u.failAnalysis(s"Table or view not found: ${u.tableName}")
--- End diff --
Also improved the error message to let users know the database is not found?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r101350313
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,18 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val tableIdentWithDb = u.tableIdentifier.copy(database = u.tableIdentifier.database
--- End diff --
please follow the existing style: https://github.com/apache/spark/pull/16674/files#diff-57b3d87be744b7d79a9beacf8e5e5eb2L622
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100002255
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -427,15 +481,15 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
test("Using view after switching current database") {
withView("v") {
- sql("CREATE VIEW v AS SELECT * FROM src")
+ sql("CREATE VIEW v AS SELECT * FROM jt")
--- End diff --
Change table `src` to `jt`, so we only have to create one single table in `beforeAll()`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100594514
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -398,26 +472,6 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
- test("create hive view for json table") {
--- End diff --
hmmm? these tests are still here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r101110989
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala ---
@@ -617,13 +617,18 @@ class Analyzer(
private def lookupTableFromCatalog(
u: UnresolvedRelation,
defaultDatabase: Option[String] = None): LogicalPlan = {
+ val db = u.tableIdentifier.database.orElse(defaultDatabase)
+ val tableIdentWithDb = u.tableIdentifier.copy(database = db)
try {
- val tableIdentWithDb = u.tableIdentifier.copy(
- database = u.tableIdentifier.database.orElse(defaultDatabase))
catalog.lookupRelation(tableIdentWithDb, u.alias)
} catch {
case _: NoSuchTableException =>
- u.failAnalysis(s"Table or view not found: ${u.tableName}")
+ u.failAnalysis(s"Table or view not found: ${tableIdentWithDb.unquotedString}")
+ // If the database is defined and that database is not found, throw an AnalysisException.
+ // Note that if the database is not defined, it is possible we are looking up a temp view.
+ case _: NoSuchDatabaseException if db.isDefined =>
--- End diff --
if users don't specify database, and the current database is removed by other sessions, then they will hit `NoSuchDatabaseException`. In this case, I think we should also throw `AnalysisException` for consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/16674
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72573 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72573/testReport)** for PR 16674 at commit [`2e201c2`](https://github.com/apache/spark/commit/2e201c26415104d4b9d417399cb87db0c8d07308).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #16674: [SPARK-19331][SQL][TESTS] Improve the test coverage of S...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/16674
**[Test build #72567 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72567/testReport)** for PR 16674 at commit [`2e201c2`](https://github.com/apache/spark/commit/2e201c26415104d4b9d417399cb87db0c8d07308).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by jiangxb1987 <gi...@git.apache.org>.
Github user jiangxb1987 commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100455122
--- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hive.execution
+
+import org.apache.spark.sql.{AnalysisException, Row, SaveMode, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
+import org.apache.spark.sql.execution.SQLViewSuite
+import org.apache.spark.sql.hive.test.{TestHive, TestHiveSingleton}
+import org.apache.spark.sql.types.StructType
+
+/**
+ * A test suite for Hive view related functionality.
+ */
+class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
+ protected override val spark: SparkSession = TestHive.sparkSession
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ // Create a simple table with two columns: id and id1
+ spark.range(1, 10).selectExpr("id", "id id1").write.format("json").saveAsTable("jt")
+ }
+
+ override def afterAll(): Unit = {
+ try {
+ spark.sql(s"DROP TABLE IF EXISTS jt")
+ } finally {
+ super.afterAll()
+ }
+ }
+
+ import testImplicits._
+
+ test("create a permanent/temp view using a hive, built-in, and permanent user function") {
+ val permanentFuncName = "myUpper"
+ val permanentFuncClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ val builtInFuncNameInLowerCase = "abs"
+ val builtInFuncNameInMixedCase = "aBs"
+ val hiveFuncName = "histogram_numeric"
+
+ withUserDefinedFunction(permanentFuncName -> false) {
+ sql(s"CREATE FUNCTION $permanentFuncName AS '$permanentFuncClass'")
+ withTable("tab1") {
+ (1 to 10).map(i => (s"$i", i)).toDF("str", "id").write.saveAsTable("tab1")
+ Seq("VIEW", "TEMPORARY VIEW").foreach { viewMode =>
+ withView("view1") {
+ sql(
+ s"""
+ |CREATE $viewMode view1
+ |AS SELECT
+ |$permanentFuncName(str),
+ |$builtInFuncNameInLowerCase(id),
+ |$builtInFuncNameInMixedCase(id) as aBs,
+ |$hiveFuncName(id, 5) over()
+ |FROM tab1
+ """.stripMargin)
+ checkAnswer(sql("select count(*) FROM view1"), Row(10))
+ }
+ }
+ }
+ }
+ }
+
+ test("create a permanent/temp view using a temporary function") {
+ val tempFunctionName = "temp"
+ val functionClass =
+ classOf[org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper].getCanonicalName
+ withUserDefinedFunction(tempFunctionName -> true) {
+ sql(s"CREATE TEMPORARY FUNCTION $tempFunctionName AS '$functionClass'")
+ withView("view1", "tempView1") {
+ withTable("tab1") {
+ (1 to 10).map(i => s"$i").toDF("id").write.saveAsTable("tab1")
+
+ // temporary view
+ sql(s"CREATE TEMPORARY VIEW tempView1 AS SELECT $tempFunctionName(id) from tab1")
+ checkAnswer(sql("select count(*) FROM tempView1"), Row(10))
+
+ // permanent view
+ val e = intercept[AnalysisException] {
+ sql(s"CREATE VIEW view1 AS SELECT $tempFunctionName(id) from tab1")
+ }.getMessage
+ assert(e.contains("Not allowed to create a permanent view `view1` by referencing " +
+ s"a temporary function `$tempFunctionName`"))
+ }
+ }
+ }
+ }
+
+ test("create hive view for json table") {
+ // json table is not hive-compatible, make sure the new flag fix it.
+ withView("testView") {
+ sql("CREATE VIEW testView AS SELECT id FROM jt")
+ checkAnswer(sql("SELECT * FROM testView ORDER BY id"), (1 to 9).map(i => Row(i)))
+ }
+ }
+
+ test("create hive view for partitioned parquet table") {
+ // partitioned parquet table is not hive-compatible, make sure the new flag fix it.
+ withTable("parTable") {
+ withView("testView") {
+ val df = Seq(1 -> "a").toDF("i", "j")
+ df.write.format("parquet").partitionBy("i").saveAsTable("parTable")
+ sql("CREATE VIEW testView AS SELECT i, j FROM parTable")
+ checkAnswer(sql("SELECT * FROM testView"), Row(1, "a"))
+ }
+ }
+ }
+
+ test("create hive view for joined tables") {
--- End diff --
Sure - let me do this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r100594045
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -334,6 +389,25 @@ class SQLViewSuite extends QueryTest with SQLTestUtils with TestHiveSingleton {
}
}
+ test("correctly handle ALTER VIEW on a referenced view") {
+ withTable("jt2") {
+ val df = (1 until 10).map(i => (i, i + 1)).toDF("i", "j")
+ df.write.format("json").saveAsTable("jt2")
+
+ withView("view1", "view2") {
+ sql("CREATE VIEW view1(x, y) AS SELECT * FROM jt")
+
+ // Create a nested view.
+ sql("CREATE VIEW view2(id, id1) AS SELECT * FROM view1")
+ checkAnswer(sql("SELECT * FROM view2 ORDER BY id"), (1 to 9).map(i => Row(i, i)))
+
+ // Alter the referenced view.
+ sql("ALTER VIEW view1 AS SELECT i AS x, j AS y FROM jt2")
--- End diff --
nit: `ALTER VIEW view1 AS SELECT id AS x, id1 + 1 As y FROM jt`, then we don't need to create `jt2`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #16674: [SPARK-19331][SQL][TESTS] Improve the test covera...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/16674#discussion_r101111083
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala ---
@@ -15,29 +15,34 @@
* limitations under the License.
*/
-package org.apache.spark.sql.hive.execution
+package org.apache.spark.sql.execution
-import org.apache.spark.sql.{AnalysisException, QueryTest, Row, SaveMode}
+import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis.NoSuchTableException
-import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType}
-import org.apache.spark.sql.hive.test.TestHiveSingleton
-import org.apache.spark.sql.test.SQLTestUtils
-import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.test.{SharedSQLContext, SQLTestUtils}
+
+class SimpleSQLViewSuite extends SQLViewSuite with SharedSQLContext {
+}
--- End diff --
the `{ }` is not needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org