You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/11/26 04:11:25 UTC
[GitHub] [spark] Peng-Lei opened a new pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Peng-Lei opened a new pull request #34719:
URL: https://github.com/apache/spark/pull/34719
### What changes were proposed in this pull request?
1. Move the `SHOW CREATE TABLE` testcase from `DDLParserSuite.scala` to `ShowCreateTableParserSuite.scala`.
2. Move the `SHOW CREATE TABLE` testcase from `DataSourceV2SQLSuite.scala` to`v2.ShowCreateTableSuite` and extract some testcases to `ShowCreateTableBaseSuite`
3. Merge some testcases into one testcase in `sql/ShowCreateTableSuite.scala` and move it to `v1.ShowCreateTableSuite`
4. Move the testcase from `HiveShowCreateTableSuite` to `hive.ShowCreateTableSuite`
5. Extract some testcase from `v1.ShowCreateTableSuite` and `hive.ShowCreateTableSuite` to `ShowCreateTableBaseSuite`
6. Move the `SHOW CREATE TABLE` testcase from `CharVarcharTestSuite.scala` and `SQLViewSuite.scala` to `ShowCreateTableBaseSuite`
The changes follow the approach of [#30287](https://github.com/apache/spark/pull/30287) [#34305](https://github.com/apache/spark/pull/34305)
### Why are the changes needed?
1. The unification will allow to run common `SHOW CREATE TABLE` tests for both DSv1/Hive DSv1 and DSv2
2. We can detect missing features and differences between DSv1 and DSv2 implementations.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowCreateTableSuite"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778891936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL(0) == s"CREATE TABLE $fullName (")
+ assert(showDDL(1) == "`a` INT,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`c` DECIMAL(2,1))")
+ assert(showDDL(4) == s"USING json")
+ assert(showDDL(5) == "PARTITIONED BY (c)")
+ assert(showDDL(6) == "CLUSTERED BY (a)")
+ assert(showDDL(7) == s"SORTED BY (b)")
+ assert(showDDL(8) == s"INTO 2 BUCKETS")
+ assert(showDDL(9) == "COMMENT 'This is a comment'")
+ assert(showDDL(10) == "TBLPROPERTIES (")
+ assert(showDDL(11) == s"'a' = '1')")
+ }
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ test("show create table") {
Review comment:
two diffs between v1 and hive:
1. default using does not match. The defaultUsing == "USING HIVE" when test for hive catalog, but the result is `USING TEXT` of `show create table`. because hive is not datasource table, so we call `convertTableMetadata` for `SHOW CREATE TABLE`. Then `HiveSerDe.serdeToSource` to get the source as the provider of output. The code is [#L1100](https://github.com/apache/spark/blob/b9b5562f6b458c59a706fd6ab0d5b2813d0bdb96/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L1100)
2. more properties(transient_lastDdlTime) for hive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778684071
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778847164
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.QueryTest
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+ protected def ns: String = "ns1"
+ protected def table: String = "tbl"
+ protected def fullName: String
+
+ test("SPARK-33892: show create table w/ char/varchar") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("SPARK-36012: add null flag when show create table") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin).show()
Review comment:
```suggestion
""".stripMargin)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r783073012
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala
##########
@@ -903,4 +903,88 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("temp view") {
Review comment:
Ah sorry I pointed to the run test suite. This is the old test suite, we have a new one for view: `SQLViewTestSuite`.
For temp view, the test can be put it `TempViewTestSuite`, which extends `SQLViewTestSuite` and runs with both local and global temp view.
For persistent view, the test can be put it `PersistedViewTestSuite`
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala
##########
@@ -903,4 +903,88 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("temp view") {
Review comment:
Ah sorry I pointed to the wrong test suite. This is the old test suite, we have a new one for view: `SQLViewTestSuite`.
For temp view, the test can be put it `TempViewTestSuite`, which extends `SQLViewTestSuite` and runs with both local and global temp view.
For persistent view, the test can be put it `PersistedViewTestSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757256714
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
diff with V2:
1.Options `\`key\`` \`value\`` instead of `'key' = 'value'`
2.Partition column is at last of columns
3.location start with `file:`
4.`TBLPROPERTIES (` insted of `TBLPROPERTIES(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758267528
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
Review comment:
for `Hive V1`. That defaultUsing == "USING HIVE", It will failed. The reason as follows:
1. Test data source does not support int data type.
2. Dynamic partition strict mode resuires at least one static partition column.
3. Create Hive serde table is not supported yet.
So I change it to "USING json". It just persisting data source table into Hive metastore in `json` which is not compatible with Hive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758154197
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
And we know that this check is done by the analyzer and has nothing to do with the underlying table/catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758235648
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c"),
+ ShowCreateTable(
+ UnresolvedTableOrView(Seq("a", "b", "c"), "SHOW CREATE TABLE", allowTempView = false)))
+
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c AS SERDE"),
Review comment:
Ok. I'm just a little bit skeptical about whether we're going to get rid of this option `SERDE` to unify this command , Because it only exists for the Hive serde table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979839415
**[Test build #145652 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145652/testReport)** for PR 34719 at commit [`22356f7`](https://github.com/apache/spark/commit/22356f775deacd193203bd6f50e72de2751f2797).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ShowCreateTableParserSuite extends AnalysisTest `
* `trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils `
* `class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase `
* `class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase `
* `class ShowCreateTableSuite extends v1.ShowCreateTableSuiteBase with CommandSuiteBase `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758183803
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
Yeah. I think our SQL parser support both syntaxes.
I write a test case for test it
```scala
test("test") {
val t = "testcat.ns1.ns2.tbl"
withTable(t) {
sql(
s"""
|CREATE TABLE $t (
| a bigint NOT NULL,
| b bigint
|)
|USING foo
|OPTIONS (
| `from` '0',
| `to` '1',
| `via` '2')
""".stripMargin)
val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
showDDL.foreach { i =>
println(i)
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778878514
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
Review comment:
```scala
sql(
s"""CREATE TABLE $t
|USING json
|PARTITIONED BY (c)
|CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
|COMMENT 'This is a comment'
|TBLPROPERTIES ('a' = '1')
|AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
""".stripMargin
)
```
v2 result is
```scala
PARTITIONED BY (c, bucket(2, a, b))
...
TBLPROPERTIES(
...
```
v1 result is :
```scala
PARTITIONED BY (c)
CLUSTERED BY (a)
SORTED BY (b)
INTO 2 BUCKETS
...
TBLPROPERTIES (
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778686393
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
done.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778686165
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778682540
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778849186
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
Review comment:
why doesn't this test work in v2?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778896399
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -843,16 +843,6 @@ class FileSourceCharVarcharTestSuite extends CharVarcharTestSuite with SharedSpa
}
}
- // TODO(SPARK-33898): Move these tests to super after SHOW CREATE TABLE for v2 implemented
Review comment:
Could you help me review the [#34773](https://github.com/apache/spark/pull/34773). In the PR. I try to fix two problem 1.Move the SHOW CREATE TABLE w/ char/varchar to CharVarcharDDLTestBase
2.Fix the behavior different with v1 command that about the TBLPROPERTIES
Thank you very much. @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r783220944
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveSQLViewSuite.scala
##########
@@ -181,4 +181,43 @@ class HiveSQLViewSuite extends SQLViewSuite with TestHiveSingleton {
}
}
}
+
+ test("hive partitioned view is not supported") {
Review comment:
@cloud-fan I didn't find the right location for this test case. So I put it here for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r785584189
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala
##########
@@ -591,4 +605,52 @@ class PersistedViewTestSuite extends SQLViewTestSuite with SharedSparkSession {
s" The view ${table.qualifiedName} may have been tampered with"))
}
}
+
+ test("show create table for persisted simple view") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a")
+ val expected = "CREATE VIEW `default`.`v1` ( `a`) AS SELECT 1 AS a"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted view with output columns") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a, 2 AS b", Seq("a", "b COMMENT 'b column'"))
+ val expected = "CREATE VIEW `default`.`v1` ( `a`, `b` COMMENT 'b column')" +
+ " AS SELECT 1 AS a, 2 AS b"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted simple view with table comment and properties") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS c1, '2' AS c2", Seq("c1 COMMENT 'bla'", "c2"),
+ Seq("COMMENT 'table comment'", "TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')"))
+
+ val expected = "CREATE VIEW `default`.`v1` ( `c1` COMMENT 'bla', `c2`)" +
+ " COMMENT 'table comment'" +
+ " TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')" +
+ " AS SELECT 1 AS c1, '2' AS c2"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
Review comment:
Made a followup: https://github.com/apache/spark/pull/35222
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979714836
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979685043
**[Test build #145651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145651/testReport)** for PR 34719 at commit [`47d5ca4`](https://github.com/apache/spark/commit/47d5ca4d4eb5d3d292de44776ff517702fa6e19b).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ShowCreateTableParserSuite extends AnalysisTest `
* `trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils `
* `class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase `
* `class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase `
* `class ShowCreateTableSuite extends v1.ShowCreateTableSuiteBase with CommandSuiteBase `
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979684305
**[Test build #145651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145651/testReport)** for PR 34719 at commit [`47d5ca4`](https://github.com/apache/spark/commit/47d5ca4d4eb5d3d292de44776ff517702fa6e19b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757256714
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
diff with V2:
1.Options ` \\`key\\` ` \\`value\\`` instead of `'key' = 'value'`
2.Partition column is at last of columns
3.location start with `file:`
4.`TBLPROPERTIES (` insted of `TBLPROPERTIES(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758157813
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
This looks like a bug in v1, as Spark silently changes the table schema from what the user specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758910060
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V2 table catalogs.
+ */
+class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase {
+ test("SPARK-33898: SHOW CREATE TABLE AS SERDE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
+ val e = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $t AS SERDE")
+ }
+ assert(e.message.contains(s"SHOW CREATE TABLE AS SERDE is not supported for v2 tables."))
+ }
+ }
+
+ test("CTAS") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |$defaultUsing
+ |PARTITIONED BY (a)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` INT,",
+ "`b` STRING)",
+ defaultUsing,
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "TBLPROPERTIES(",
+ "'a' = '1')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
Review comment:
I think [#34704](https://github.com/apache/spark/pull/34704) maybe just qualify location for v2 command about namespace action. In v2 table location is as a property of table. we should qualify location when create table if table properties contains location key. I want do it. agree?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-995583431
@cloud-fan en. I've been dealing with other things a while ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758156350
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
I think the v2 version is better. Does our SQL parser support both syntaxes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758154600
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
Review comment:
does this work with v2 catalogs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758953291
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
Review comment:
I guess `CATS` is testing `Create Table As Select`？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759173603
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
The issue will be resolved by [#34753](https://github.com/apache/spark/pull/34753)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #34719:
URL: https://github.com/apache/spark/pull/34719
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r782769667
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+
+ test("show create table[simple]") {
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |using parquet
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(t)
+ assert(showDDL === Array(
+ s"CREATE TABLE $fullName (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ "USING parquet",
+ "OPTIONS (",
+ "'from' = '0',",
+ "'to' = '1',",
+ "'via' = '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
+ }
+
+ test("bucketed data source table") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val expected = s"CREATE TABLE $fullName ( `a` INT, `b` STRING) USING json" +
+ s" CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS"
+ assert(getShowCreateDDL(t).mkString(" ") == expected)
+ }
+ }
+
+ test("partitioned bucketed data source table") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val expected = s"CREATE TABLE $fullName ( `a` INT, `b` STRING, `c` DECIMAL(2,1)) USING json" +
+ s" PARTITIONED BY (c) CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS"
+ assert(getShowCreateDDL(t).mkString(" ") == expected)
+ }
+ }
+
+ test("temp view") {
Review comment:
can we move the test to `SQLViewSuite` to test all view types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r782772057
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,215 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+
+ test("show create table[simple]") {
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |using parquet
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(t)
+ assert(showDDL === Array(
+ s"CREATE TABLE $fullName (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ "USING parquet",
+ "OPTIONS (",
+ "'from' = '0',",
+ "'to' = '1',",
+ "'via' = '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
+ }
+
+ test("bucketed data source table") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val expected = s"CREATE TABLE $fullName ( `a` INT, `b` STRING) USING json" +
+ s" CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS"
+ assert(getShowCreateDDL(t).mkString(" ") == expected)
+ }
+ }
+
+ test("partitioned bucketed data source table") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val expected = s"CREATE TABLE $fullName ( `a` INT, `b` STRING, `c` DECIMAL(2,1)) USING json" +
+ s" PARTITIONED BY (c) CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS"
+ assert(getShowCreateDDL(t).mkString(" ") == expected)
+ }
+ }
+
+ test("temp view") {
Review comment:
ok, I will do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r785579900
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala
##########
@@ -591,4 +605,52 @@ class PersistedViewTestSuite extends SQLViewTestSuite with SharedSparkSession {
s" The view ${table.qualifiedName} may have been tampered with"))
}
}
+
+ test("show create table for persisted simple view") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a")
+ val expected = "CREATE VIEW `default`.`v1` ( `a`) AS SELECT 1 AS a"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted view with output columns") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a, 2 AS b", Seq("a", "b COMMENT 'b column'"))
+ val expected = "CREATE VIEW `default`.`v1` ( `a`, `b` COMMENT 'b column')" +
+ " AS SELECT 1 AS a, 2 AS b"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted simple view with table comment and properties") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS c1, '2' AS c2", Seq("c1 COMMENT 'bla'", "c2"),
+ Seq("COMMENT 'table comment'", "TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')"))
+
+ val expected = "CREATE VIEW `default`.`v1` ( `c1` COMMENT 'bla', `c2`)" +
+ " COMMENT 'table comment'" +
+ " TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')" +
+ " AS SELECT 1 AS c1, '2' AS c2"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
Review comment:
Hm, this test is flaky:
```
sbt.ForkMain$ForkError: org.scalatest.exceptions.TestFailedException: "...BLPROPERTIES ( 'prop[2' = 'value2', 'prop1' = 'value1]') AS SELECT 1 AS c1..." did not equal "...BLPROPERTIES ( 'prop[1' = 'value1', 'prop2' = 'value2]') AS SELECT 1 AS c1..."
at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)
at org.apache.spark.sql.execution.PersistedViewTestSuite.$anonfun$new$139(SQLViewTestSuite.scala:643)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1475)
at org.apache.spark.sql.test.SQLTestUtilsBase.withView(SQLTestUtils.scala:317)
at org.apache.spark.sql.test.SQLTestUtilsBase.withView$(SQLTestUtils.scala:315)
at org.apache.spark.sql.execution.SQLViewTestSuite.withView(SQLViewTestSuite.scala:37)
at org.apache.spark.sql.execution.PersistedViewTestSuite.$anonfun$new$138(SQLViewTestSuite.scala:635)
at org.apache.spark.sql.execution.PersistedViewTestSuite.$anonfun$new$138$adapted(SQLViewTestSuite.scala:634)
at scala.collection.immutable.List.foreach(List.scala:333)
at org.apache.spark.sql.execution.PersistedViewTestSuite.$anonfun$new$137(SQLViewTestSuite.scala:634)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
at org.scalatest.Transformer.apply(Transformer.scala:22)
at org.scalatest.Transformer.apply(Transformer.scala:20)
at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:190)
at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:203)
at org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:188)
at org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:200)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-995125557
can we continue this PR now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778700123
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c"),
+ ShowCreateTable(
+ UnresolvedTableOrView(Seq("a", "b", "c"), "SHOW CREATE TABLE", allowTempView = false)))
+
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c AS SERDE"),
Review comment:
@MaxGekk `AS SERDE` just for get the hive ddl not for spark ddl. if we do not use `as serde`, we also try to convert it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778878514
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
Review comment:
```scala
sql(
s"""CREATE TABLE $t
|USING json
|PARTITIONED BY (c)
|CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
|COMMENT 'This is a comment'
|TBLPROPERTIES ('a' = '1')
|AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
""".stripMargin
)
```
v2 result is
```scala
PARTITIONED BY (c, bucket(2, a, b))
...
TBLPROPERTIES(
...
```
v1 and hive result is
```scala
PARTITIONED BY (c)
CLUSTERED BY (a)
SORTED BY (b)
INTO 2 BUCKETS
...
TBLPROPERTIES (
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979700216
**[Test build #145652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145652/testReport)** for PR 34719 at commit [`22356f7`](https://github.com/apache/spark/commit/22356f775deacd193203bd6f50e72de2751f2797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759172707
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V2 table catalogs.
+ */
+class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase {
+ test("SPARK-33898: SHOW CREATE TABLE AS SERDE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
+ val e = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $t AS SERDE")
+ }
+ assert(e.message.contains(s"SHOW CREATE TABLE AS SERDE is not supported for v2 tables."))
+ }
+ }
+
+ test("CTAS") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |$defaultUsing
+ |PARTITIONED BY (a)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` INT,",
+ "`b` STRING)",
+ defaultUsing,
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "TBLPROPERTIES(",
+ "'a' = '1')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
Review comment:
Thank you very much. The issue will be resolved by [#34758](https://github.com/apache/spark/pull/34758)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759810223
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
Review comment:
Thank you. That's true, you are right.
```scala
withTable(t) {
...
checkCreateTableOrView // without drop command
}
```
seems more safe. I will modify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei edited a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei edited a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-1010689654
do sumary
Test case:
1. Move the testcase from `DDLParserSuite.scala` to `ShowCreateTableParserSuite.scala`.
2. Move the testcase from `DataSourceV2SQLSuite` to `v2.ShowCreateTableSuite`.
If the test case also work with V1/Hive, then move to ShowCreateTableSuiteBase
3. Move the testcase from `sql/ShowCreateTableSuite.scala` to `v1.ShowCreateTableSuite`
If the test case also work with Hive, then move to `v1.ShowCreateTableSuiteBase`
If the test case also work with V2/Hive, then move to `ShowCreateTableSuiteBase`
4. Move the testcase from `HiveShowCreateTableSuite` to `hive.ShowCreateTableSuite`
If the test case also work with V1/V2, then move to `ShowCreateTableSuiteBase`
If the test case also work with V1, then move to `v1.ShowCreateTableSuiteBase`
5. Use `getShowCreateDDL` instead of `checkCreateTable` to check the result.
code:
1. Add one space after `create table xxx` for the command `SHOW CREATE TABLE AS SERDE` to unify the output.
2. Add one space after `OPTIONS` for v2 command `SHOW CREATE TABLE` to unify the output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-1010689654
do sumary
Test case:
1.Move the testcase from `DDLParserSuite.scala` to `ShowCreateTableParserSuite.scala`.
2.Move the testcase from `DataSourceV2SQLSuite` to `v2.ShowCreateTableSuite`.
If the test case also work with V1/Hive, then move to ShowCreateTableSuiteBase
3.Move the testcase from `sql/ShowCreateTableSuite.scala` to `v1.ShowCreateTableSuite`
If the test case also work with Hive, then move to `v1.ShowCreateTableSuiteBase`
If the test case also work with V2/Hive, then move to `ShowCreateTableSuiteBase`
4.Move the testcase from `HiveShowCreateTableSuite` to `hive.ShowCreateTableSuite`
If the test case also work with V1/V2, then move to `ShowCreateTableSuiteBase`
If the test case also work with V1, then move to `v1.ShowCreateTableSuiteBase`
5.Use `getShowCreateDDL` instead of `checkCreateTable` to check the result.
code:
1.Add one space after `create table xxx` for the command `SHOW CREATE TABLE AS SERDE` to unify the output.
2.Add one space after `OPTIONS` for v2 command `SHOW CREATE TABLE` to unify the output
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757251460
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/ShowCreateTableSuite.scala
##########
@@ -1,241 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql
-
-import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.sources.SimpleInsertSource
-import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
-import org.apache.spark.util.Utils
-
-class SimpleShowCreateTableSuite extends ShowCreateTableSuite with SharedSparkSession
-
-abstract class ShowCreateTableSuite extends QueryTest with SQLTestUtils {
- import testImplicits._
-
- test("data source table with user specified schema") {
- withTable("ddl_test") {
- val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
-
- sql(
- s"""CREATE TABLE ddl_test (
- | a STRING,
- | b STRING,
- | `extra col` ARRAY<INT>,
- | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
- |)
- |USING json
- |OPTIONS (
- | PATH '$jsonFilePath'
- |)
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table CTAS") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (b)
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (c)
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with a comment") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |COMMENT 'This is a comment'
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with table properties") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |TBLPROPERTIES ('a' = '1')
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
Review comment:
Merge into one testcase `test("CATS")` in `ShowCreateTableBaseSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r783569453
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
Review comment:
@cloud-fan When use V2 command default, Do you think we should change V1 command behavior to match V2 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778846637
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/CharVarcharTestSuite.scala
##########
@@ -843,16 +843,6 @@ class FileSourceCharVarcharTestSuite extends CharVarcharTestSuite with SharedSpa
}
}
- // TODO(SPARK-33898): Move these tests to super after SHOW CREATE TABLE for v2 implemented
Review comment:
The TODO says we should move it to the super class `CharVarcharTestSuite`, not `ShowCreateTableSuiteBase`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778891936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL(0) == s"CREATE TABLE $fullName (")
+ assert(showDDL(1) == "`a` INT,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`c` DECIMAL(2,1))")
+ assert(showDDL(4) == s"USING json")
+ assert(showDDL(5) == "PARTITIONED BY (c)")
+ assert(showDDL(6) == "CLUSTERED BY (a)")
+ assert(showDDL(7) == s"SORTED BY (b)")
+ assert(showDDL(8) == s"INTO 2 BUCKETS")
+ assert(showDDL(9) == "COMMENT 'This is a comment'")
+ assert(showDDL(10) == "TBLPROPERTIES (")
+ assert(showDDL(11) == s"'a' = '1')")
+ }
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ test("show create table") {
Review comment:
two diffs between v1 and hive:
1. default using does not match. The defaultUsing == "USING HIVE" when test for hive catalog, but the result is `USING TEXT` of `show create table`. because hive is not datasource table, so we call `convertTableMetadata` for `SHOW CREATE TABLE`. Then `HiveSerDe.serdeToSource` to get the source as the provider of output.
2. more properties(transient_lastDdlTime) for hive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758174307
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
Review comment:
Yeah. I test it. It use `test_catalog`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758208485
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
let's have a new PR to fix the v1 SHOW CREATE TABLE then
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758966706
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
Review comment:
Sorry. Wrong title. I did mean to express `Create Table As Select`. It should be `CTAS` instead of `CATS`. `CATS` is short name of `Contention-Aware Transaction Scheduling` in MySQL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758966959
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
Review comment:
I will address the comments together later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757257983
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V2 table catalogs.
+ */
+class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase {
+ test("SPARK-33898: SHOW CREATE TABLE AS SERDE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
+ val e = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $t AS SERDE")
+ }
+ assert(e.message.contains(s"SHOW CREATE TABLE AS SERDE is not supported for v2 tables."))
+ }
+ }
+
+ test("CTAS") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |$defaultUsing
+ |PARTITIONED BY (a)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` INT,",
+ "`b` STRING)",
+ defaultUsing,
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "TBLPROPERTIES(",
+ "'a' = '1')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` BIGINT NOT NULL,",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)",
+ defaultUsing,
+ "OPTIONS(",
+ "'from' = '0',",
+ "'to' = '1',",
+ "'via' = '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION '/tmp'",
+ "TBLPROPERTIES(",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE PARTITIONED BY TRANSFORMS") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (a INT, b STRING, ts TIMESTAMP) $defaultUsing
+ |PARTITIONED BY (
+ | a,
+ | bucket(16, b),
+ | years(ts),
+ | months(ts),
+ | days(ts),
+ | hours(ts))
+ """.stripMargin)
+ // V1 transforms cannot be converted to partition columns: bucket,year,...)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
Review comment:
As we could not convert to patition columns for v1,so put it here just for v2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757256714
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
diff with V2:
1.Options ` key` `value` instead of `'key' = 'value'`
2.Partition column is at last of columns
3.location start with `file:`
4.`TBLPROPERTIES (` insted of `TBLPROPERTIES(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979844618
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145652/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-995583431
@cloud-fan en. I've been dealing with other things a while ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r785585691
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala
##########
@@ -591,4 +605,52 @@ class PersistedViewTestSuite extends SQLViewTestSuite with SharedSparkSession {
s" The view ${table.qualifiedName} may have been tampered with"))
}
}
+
+ test("show create table for persisted simple view") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a")
+ val expected = "CREATE VIEW `default`.`v1` ( `a`) AS SELECT 1 AS a"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted view with output columns") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a, 2 AS b", Seq("a", "b COMMENT 'b column'"))
+ val expected = "CREATE VIEW `default`.`v1` ( `a`, `b` COMMENT 'b column')" +
+ " AS SELECT 1 AS a, 2 AS b"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted simple view with table comment and properties") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS c1, '2' AS c2", Seq("c1 COMMENT 'bla'", "c2"),
+ Seq("COMMENT 'table comment'", "TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')"))
+
+ val expected = "CREATE VIEW `default`.`v1` ( `c1` COMMENT 'bla', `c2`)" +
+ " COMMENT 'table comment'" +
+ " TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')" +
+ " AS SELECT 1 AS c1, '2' AS c2"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
Review comment:
thank you fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757251460
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/ShowCreateTableSuite.scala
##########
@@ -1,241 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql
-
-import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.sources.SimpleInsertSource
-import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
-import org.apache.spark.util.Utils
-
-class SimpleShowCreateTableSuite extends ShowCreateTableSuite with SharedSparkSession
-
-abstract class ShowCreateTableSuite extends QueryTest with SQLTestUtils {
- import testImplicits._
-
- test("data source table with user specified schema") {
- withTable("ddl_test") {
- val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
-
- sql(
- s"""CREATE TABLE ddl_test (
- | a STRING,
- | b STRING,
- | `extra col` ARRAY<INT>,
- | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
- |)
- |USING json
- |OPTIONS (
- | PATH '$jsonFilePath'
- |)
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table CTAS") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (b)
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (c)
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with a comment") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |COMMENT 'This is a comment'
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with table properties") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |TBLPROPERTIES ('a' = '1')
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
Review comment:
Merge into one testcase `test("CATS")` in `v1.ShowCreateTableSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979740143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758189709
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c"),
+ ShowCreateTable(
+ UnresolvedTableOrView(Seq("a", "b", "c"), "SHOW CREATE TABLE", allowTempView = false)))
+
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c AS SERDE"),
Review comment:
The doc https://github.com/apache/spark/blob/master/docs/sql-ref-syntax-aux-show-create-table.md doesn't describe the `SERDE` option. Could you open an JIRA if such ticket doesn't exist.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
Review comment:
Just inline `db` and `table` into `withNamespaceAndTable` since they are used only once.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
Why it is in upper case? Could you align test titles to other unified test suites
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
Review comment:
Can you incapsulate catalog specific things in separate traits.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` BIGINT NOT NULL,")
+ assert(showDDL(2) == "`b` BIGINT)")
+ assert(showDDL(3) == s"USING ${classOf[SimpleInsertSource].getName}")
+ }
+ }
+
+ test("SPARK-24911: KEEP QUOTES FOR NESTED FIELDS") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t (`a` STRUCT<`b`: STRING>) $defaultUsing")
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRUCT<`b`: STRING>)")
+ if (catalogVersion == "Hive V1") {
Review comment:
If there are diffs, please, put tests to the catalog specific test suites. This will help to unify command's output in the future otherwise we will need to review all base test suite and find all such `if`s
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
Review comment:
`CATS`? What does the test check? Should be clear looking at the test title.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
Please, place the diffs to catalog specific test suites instead of describing the diffs in text. Also, open JIRAs and add TODOs to fix the differences.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
Review comment:
Why not $defaultUsing?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` BIGINT NOT NULL,")
+ assert(showDDL(2) == "`b` BIGINT)")
+ assert(showDDL(3) == s"USING ${classOf[SimpleInsertSource].getName}")
+ }
+ }
+
+ test("SPARK-24911: KEEP QUOTES FOR NESTED FIELDS") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t (`a` STRUCT<`b`: STRING>) $defaultUsing")
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRUCT<`b`: STRING>)")
+ if (catalogVersion == "Hive V1") {
+ assert(showDDL(2) == "USING text")
+ } else {
+ assert(showDDL(2) == defaultUsing)
+ }
+ }
+ }
+
+ test("DATA SOURCE TABLE WITH USER SPECIFIED SCHEMA") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
+ sql(
+ s"""CREATE TABLE $t (
+ | a STRING,
+ | b STRING,
+ | `extra col` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |USING json
+ |OPTIONS (
+ | PATH '$jsonFilePath'
+ |)
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRING,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`extra col` ARRAY<INT>,")
+ assert(showDDL(4) == "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)")
+ assert(showDDL(5) == "USING json")
+ // V2 showDDL(6) == LOCATION 'jsonFilePath'
+ // V1 showDDL(6) == LOCATION 'file:jsonFilePath'
+ }
+ }
+
+ def getShowCreateDDL(showCreateTableSql: String): Array[String] = {
+ sql(showCreateTableSql).head().getString(0).split("\n").map(_.trim)
+ }
Review comment:
How about inverse way: take an expected output and compare it to command output?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
Review comment:
withTable has more complex logic in handling exceptions, see:
```scala
protected def withTable(tableNames: String*)(f: => Unit): Unit = {
Utils.tryWithSafeFinally(f) {
tableNames.foreach { name =>
spark.sql(s"DROP TABLE IF EXISTS $name")
}
}
}
```
https://github.com/apache/spark/blob/c6c72a453b9958c32a16be3f199e256dc7bc17ae/core/src/main/scala/org/apache/spark/util/Utils.scala#L1465-L1485
Are you sure that your custom code is reliable enough?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
Review comment:
Use withNamespaceAndTable
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
Review comment:
Use "ns1" and "tbl" directly since they are used only once.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
Review comment:
Do you really test all catalogs in such way?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` BIGINT NOT NULL,")
+ assert(showDDL(2) == "`b` BIGINT)")
+ assert(showDDL(3) == s"USING ${classOf[SimpleInsertSource].getName}")
+ }
+ }
+
+ test("SPARK-24911: KEEP QUOTES FOR NESTED FIELDS") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t (`a` STRUCT<`b`: STRING>) $defaultUsing")
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRUCT<`b`: STRING>)")
+ if (catalogVersion == "Hive V1") {
+ assert(showDDL(2) == "USING text")
+ } else {
+ assert(showDDL(2) == defaultUsing)
+ }
+ }
+ }
+
+ test("DATA SOURCE TABLE WITH USER SPECIFIED SCHEMA") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
+ sql(
+ s"""CREATE TABLE $t (
+ | a STRING,
+ | b STRING,
+ | `extra col` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |USING json
+ |OPTIONS (
+ | PATH '$jsonFilePath'
+ |)
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRING,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`extra col` ARRAY<INT>,")
+ assert(showDDL(4) == "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)")
+ assert(showDDL(5) == "USING json")
+ // V2 showDDL(6) == LOCATION 'jsonFilePath'
+ // V1 showDDL(6) == LOCATION 'file:jsonFilePath'
Review comment:
Can you write a function which accept one expected string, splits it to lines and compares to the command output?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` BIGINT NOT NULL,")
+ assert(showDDL(2) == "`b` BIGINT)")
+ assert(showDDL(3) == s"USING ${classOf[SimpleInsertSource].getName}")
+ }
+ }
+
+ test("SPARK-24911: KEEP QUOTES FOR NESTED FIELDS") {
Review comment:
Why is the test title in upper case?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
Review comment:
Use `withNamespaceAndTable`, so, we could move it the base test trait in the future w/o any modifications.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
Review comment:
The table should exist before the check otherwise other test leaves garbage which can be considered as a bug. Please, remove the `DROP ..` command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759811062
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
Review comment:
No, I'll try to use existing framework code instead of my custom code. Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758155495
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
Review comment:
I will put this testcase into `CharVarcharDDLTestBase` later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758212663
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-995584072
> can we continue this PR now?
I've been dealing with other things a while ago. I will continue to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778685920
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
@cloud-fan This problem may be a bit slower to process for me, I want to process this PR first and I also add todo in the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778851129
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowCreateTableSuite.scala
##########
@@ -15,51 +15,23 @@
* limitations under the License.
*/
-package org.apache.spark.sql.hive
+package org.apache.spark.sql.hive.execution.command
-import org.apache.spark.sql.{AnalysisException, ShowCreateTableSuite}
+import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.hive.test.TestHiveSingleton
-import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
+import org.apache.spark.sql.execution.command.v1
+import org.apache.spark.sql.hive.HiveExternalCatalog
+import org.apache.spark.sql.internal.HiveSerDe
-class HiveShowCreateTableSuite extends ShowCreateTableSuite with TestHiveSingleton {
-
- private var origCreateHiveTableConfig = false
-
- protected override def beforeAll(): Unit = {
- super.beforeAll()
- origCreateHiveTableConfig =
- spark.conf.get(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT)
- spark.conf.set(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key, true)
- }
-
- protected override def afterAll(): Unit = {
- spark.conf.set(
- SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key,
- origCreateHiveTableConfig)
- super.afterAll()
- }
-
- test("view") {
- Seq(true, false).foreach { serde =>
- withView("v1") {
- sql("CREATE VIEW v1 AS SELECT 1 AS a")
- checkCreateView("v1", serde)
- }
- }
- }
-
- test("view with output columns") {
- Seq(true, false).foreach { serde =>
- withView("v1") {
- sql("CREATE VIEW v1 (a, b COMMENT 'b column') AS SELECT 1 AS a, 2 AS b")
- checkCreateView("v1", serde)
- }
- }
- }
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 Hive external
+ * table catalog.
+ */
+class ShowCreateTableSuite extends v1.ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
- test("view with table comment and properties") {
+ test("PERSISTED VIEW") {
Review comment:
Can you confirm that the test cases in this file only work in Hive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778695940
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` BIGINT NOT NULL,")
+ assert(showDDL(2) == "`b` BIGINT)")
+ assert(showDDL(3) == s"USING ${classOf[SimpleInsertSource].getName}")
+ }
+ }
+
+ test("SPARK-24911: KEEP QUOTES FOR NESTED FIELDS") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t (`a` STRUCT<`b`: STRING>) $defaultUsing")
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRUCT<`b`: STRING>)")
+ if (catalogVersion == "Hive V1") {
+ assert(showDDL(2) == "USING text")
+ } else {
+ assert(showDDL(2) == defaultUsing)
+ }
+ }
+ }
+
+ test("DATA SOURCE TABLE WITH USER SPECIFIED SCHEMA") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
+ sql(
+ s"""CREATE TABLE $t (
+ | a STRING,
+ | b STRING,
+ | `extra col` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |USING json
+ |OPTIONS (
+ | PATH '$jsonFilePath'
+ |)
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
+ assert(showDDL(1) == "`a` STRING,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`extra col` ARRAY<INT>,")
+ assert(showDDL(4) == "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>)")
+ assert(showDDL(5) == "USING json")
+ // V2 showDDL(6) == LOCATION 'jsonFilePath'
+ // V1 showDDL(6) == LOCATION 'file:jsonFilePath'
+ }
+ }
+
+ def getShowCreateDDL(showCreateTableSql: String): Array[String] = {
+ sql(showCreateTableSql).head().getString(0).split("\n").map(_.trim)
+ }
Review comment:
some case: `assert(showDDL(6).startsWith("LOCATION 'file:") && showDDL(6).endsWith("sample.json'"))`. maybe the current comparison method will be suitable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979685057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757251460
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/ShowCreateTableSuite.scala
##########
@@ -1,241 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql
-
-import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.sources.SimpleInsertSource
-import org.apache.spark.sql.test.{SharedSparkSession, SQLTestUtils}
-import org.apache.spark.util.Utils
-
-class SimpleShowCreateTableSuite extends ShowCreateTableSuite with SharedSparkSession
-
-abstract class ShowCreateTableSuite extends QueryTest with SQLTestUtils {
- import testImplicits._
-
- test("data source table with user specified schema") {
- withTable("ddl_test") {
- val jsonFilePath = Utils.getSparkClassLoader.getResource("sample.json").getFile
-
- sql(
- s"""CREATE TABLE ddl_test (
- | a STRING,
- | b STRING,
- | `extra col` ARRAY<INT>,
- | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
- |)
- |USING json
- |OPTIONS (
- | PATH '$jsonFilePath'
- |)
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table CTAS") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (b)
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("partitioned bucketed data source table") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |PARTITIONED BY (c)
- |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with a comment") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |COMMENT 'This is a comment'
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
-
- test("data source table with table properties") {
- withTable("ddl_test") {
- sql(
- s"""CREATE TABLE ddl_test
- |USING json
- |TBLPROPERTIES ('a' = '1')
- |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
- """.stripMargin
- )
-
- checkCreateTable("ddl_test")
- }
- }
Review comment:
Merge into one testcase `test("CATS")` in `ShowCreateTableBaseSuite`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979737271
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758970183
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V2 table catalogs.
+ */
+class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase {
+ test("SPARK-33898: SHOW CREATE TABLE AS SERDE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
+ val e = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $t AS SERDE")
+ }
+ assert(e.message.contains(s"SHOW CREATE TABLE AS SERDE is not supported for v2 tables."))
+ }
+ }
+
+ test("CTAS") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |$defaultUsing
+ |PARTITIONED BY (a)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` INT,",
+ "`b` STRING)",
+ defaultUsing,
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "TBLPROPERTIES(",
+ "'a' = '1')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
Review comment:
yes please.
Let's fix all the issues found in this PR first, then we can come back and merge this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758158177
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
@Peng-Lei can you confirm with the real table schema? maybe it's just a bug in SHOW CREATE TABLE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758153605
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
I think this test is better to be in the view suite, as we can run it with different view types automatically.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759197796
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
@cloud-fan I add code after create table
```scala
val actual = spark.sharedState.externalCatalog.getTable("default", t)
println(actual)
```
The result is as follows:
```scala
CatalogTable(
Database: default
Table: tbl
Created Time: Tue Nov 30 03:34:05 PST 2021
Last Access: UNKNOWN
Created By: Spark 3.3.0-SNAPSHOT
Type: EXTERNAL
Provider: parquet
Comment: This is a comment
Table Properties: [prop1=1, prop2=2, prop3=3, prop4=4]
Location: file:/tmp
Storage Properties: [from=0, to=1, via=2]
Partition Provider: Catalog
Partition Columns: [`a`]
Schema: root
|-- b: long (nullable = true)
|-- c: long (nullable = true)
|-- extraCol: array (nullable = true)
| |-- element: integer (containsNull = true)
|-- <another>: struct (nullable = true)
| |-- x: integer (nullable = true)
| |-- y: array (nullable = true)
| | |-- element: boolean (containsNull = true)
|-- a: long (nullable = false)
)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-1011735454
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778690967
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
Review comment:
@MaxGekk I agree with you very much. The check function The code is the original for v1 and hive. At now. I modify the check logic to compare the ddl for v1 and v2. The hive is left, the logic is compare the catalogtable after normalize between first create table and the second table created by `showDDL` of first create table. So I left the check logic for hive. Do you agree?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778682342
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
@MaxGekk diff1 and diff 3 are already resolved. diff 2 will be resolved by [#34780](https://github.com/apache/spark/pull/34780). diff 4 will be resolved by [#34773](https://github.com/apache/spark/pull/34773)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778700123
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c"),
+ ShowCreateTable(
+ UnresolvedTableOrView(Seq("a", "b", "c"), "SHOW CREATE TABLE", allowTempView = false)))
+
+ comparePlans(
+ parsePlan("SHOW CREATE TABLE a.b.c AS SERDE"),
Review comment:
@MaxGekk `AS SERDE` just for get the hive ddl not for spark ddl. if we do not use `as serde`, we also try to convert it. The new [PR](https://github.com/apache/spark/pull/35107)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778891936
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL(0) == s"CREATE TABLE $fullName (")
+ assert(showDDL(1) == "`a` INT,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`c` DECIMAL(2,1))")
+ assert(showDDL(4) == s"USING json")
+ assert(showDDL(5) == "PARTITIONED BY (c)")
+ assert(showDDL(6) == "CLUSTERED BY (a)")
+ assert(showDDL(7) == s"SORTED BY (b)")
+ assert(showDDL(8) == s"INTO 2 BUCKETS")
+ assert(showDDL(9) == "COMMENT 'This is a comment'")
+ assert(showDDL(10) == "TBLPROPERTIES (")
+ assert(showDDL(11) == s"'a' = '1')")
+ }
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ test("show create table") {
Review comment:
two diffs between v1 and hive:
1. default using does not match. The defaultUsing == "USING HIVE" when test for hive catalog, but the result is `USING TEXT` of `show create table`. because hive is not datasource table, so we call `convertTableMetadata` for `SHOW CREATE TABLE`. Then `HiveSerDe.serdeToSource` to get the source as the provider of output.
2. more properties(transient_lastDdlTime) for hive. The code is [#L1100](https://github.com/apache/spark/blob/b9b5562f6b458c59a706fd6ab0d5b2813d0bdb96/sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala#L1100)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r778850015
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+ override def fullName: String = s"`$ns`.`$table`"
+ test("CTAS") {
+ withNamespaceAndTable(ns, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL(0) == s"CREATE TABLE $fullName (")
+ assert(showDDL(1) == "`a` INT,")
+ assert(showDDL(2) == "`b` STRING,")
+ assert(showDDL(3) == "`c` DECIMAL(2,1))")
+ assert(showDDL(4) == s"USING json")
+ assert(showDDL(5) == "PARTITIONED BY (c)")
+ assert(showDDL(6) == "CLUSTERED BY (a)")
+ assert(showDDL(7) == s"SORTED BY (b)")
+ assert(showDDL(8) == s"INTO 2 BUCKETS")
+ assert(showDDL(9) == "COMMENT 'This is a comment'")
+ assert(showDDL(10) == "TBLPROPERTIES (")
+ assert(showDDL(11) == s"'a' = '1')")
+ }
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+ // todo After SPARK-37517 unify the testcase both v1 and v2
+ test("show create table") {
Review comment:
Why doesn't this test work in hive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r783217601
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewSuite.scala
##########
@@ -903,4 +903,88 @@ abstract class SQLViewSuite extends QueryTest with SQLTestUtils {
}
}
}
+
+ test("temp view") {
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r782752672
##########
File path: sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/command/ShowCreateTableSuite.scala
##########
@@ -15,51 +15,23 @@
* limitations under the License.
*/
-package org.apache.spark.sql.hive
+package org.apache.spark.sql.hive.execution.command
-import org.apache.spark.sql.{AnalysisException, ShowCreateTableSuite}
+import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.catalog.CatalogTable
-import org.apache.spark.sql.hive.test.TestHiveSingleton
-import org.apache.spark.sql.internal.{HiveSerDe, SQLConf}
+import org.apache.spark.sql.execution.command.v1
+import org.apache.spark.sql.hive.HiveExternalCatalog
+import org.apache.spark.sql.internal.HiveSerDe
-class HiveShowCreateTableSuite extends ShowCreateTableSuite with TestHiveSingleton {
-
- private var origCreateHiveTableConfig = false
-
- protected override def beforeAll(): Unit = {
- super.beforeAll()
- origCreateHiveTableConfig =
- spark.conf.get(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT)
- spark.conf.set(SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key, true)
- }
-
- protected override def afterAll(): Unit = {
- spark.conf.set(
- SQLConf.LEGACY_CREATE_HIVE_TABLE_BY_DEFAULT.key,
- origCreateHiveTableConfig)
- super.afterAll()
- }
-
- test("view") {
- Seq(true, false).foreach { serde =>
- withView("v1") {
- sql("CREATE VIEW v1 AS SELECT 1 AS a")
- checkCreateView("v1", serde)
- }
- }
- }
-
- test("view with output columns") {
- Seq(true, false).foreach { serde =>
- withView("v1") {
- sql("CREATE VIEW v1 (a, b COMMENT 'b column') AS SELECT 1 AS a, 2 AS b")
- checkCreateView("v1", serde)
- }
- }
- }
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 Hive external
+ * table catalog.
+ */
+class ShowCreateTableSuite extends v1.ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
- test("view with table comment and properties") {
+ test("PERSISTED VIEW") {
Review comment:
1. `sql/ShowCreateTableSuite.scala` move to `v1.ShowCreateTableSuiteBase` to test for V1/Hive. if the test case also work for V2. then move to `command.ShowCreateTableSuiteBase`
2. Most of `HiveShowCreateTableSuite.scala` test case only work in Hive. So move to `hive.ShowCreateTableSuite`, try to test every test case whether work in V2/V1, If it works ok for V2/V1, then move to base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979685057
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979700216
**[Test build #145652 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145652/testReport)** for PR 34719 at commit [`22356f7`](https://github.com/apache/spark/commit/22356f775deacd193203bd6f50e72de2751f2797).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758156938
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.execution.command
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V2 table catalogs.
+ */
+class ShowCreateTableSuite extends command.ShowCreateTableSuiteBase with CommandSuiteBase {
+ test("SPARK-33898: SHOW CREATE TABLE AS SERDE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ spark.sql(s"CREATE TABLE $t (id bigint, data string) $defaultUsing")
+ val e = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $t AS SERDE")
+ }
+ assert(e.message.contains(s"SHOW CREATE TABLE AS SERDE is not supported for v2 tables."))
+ }
+ }
+
+ test("CTAS") {
+ val db = "ns1"
+ val table = "ddl_test"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""CREATE TABLE $t
+ |$defaultUsing
+ |PARTITIONED BY (a)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b
+ """.stripMargin
+ )
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE $t (",
+ "`a` INT,",
+ "`b` STRING)",
+ defaultUsing,
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "TBLPROPERTIES(",
+ "'a' = '1')"
+ ))
+ }
+ }
+
+ test("SPARK-33898: SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
Review comment:
@imback82 I thought we have fixed the path qualification bug for all v2 commands, but seems this one is missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758156771
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
Review comment:
> And we know that this check is done by the analyzer and has nothing to do with the underlying table/catalog.
make sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759200640
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
Do you think this is a bug in v1 with wrong order of columns in schema? If so, I want to try to resolve it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758162672
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-981394909
@Peng-Lei Thank you for the ping. Will review this PR today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979684305
**[Test build #145651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/145651/testReport)** for PR 34719 at commit [`47d5ca4`](https://github.com/apache/spark/commit/47d5ca4d4eb5d3d292de44776ff517702fa6e19b).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979732281
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50123/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979730567
@MaxGekk @cloud-fan @imback82 Could you take a look? Thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979716499
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50123/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757256714
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
diff with V2:
1.Options `\`key\` \`value\`` instead of `'key' = 'value'`
2.Partition column is at last of columns
3.location start with `file:`
4.`TBLPROPERTIES (` insted of `TBLPROPERTIES(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757256714
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
+ "PARTITIONED BY (a)",
+ "COMMENT 'This is a comment'",
+ "LOCATION 'file:/tmp'",
+ "TBLPROPERTIES (",
+ "'prop1' = '1',",
+ "'prop2' = '2',",
+ "'prop3' = '3',",
+ "'prop4' = '4')"
+ ))
+ }
Review comment:
diff with V2:
1.Options ` \`key\` ` \`value\`` instead of `'key' = 'value'`
2.Partition column is at last of columns
3.location start with `file:`
4.`TBLPROPERTIES (` insted of `TBLPROPERTIES(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r757254642
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
Review comment:
As could not convert `CLUSTERTED` `SORTED` into transform, So we could not put it to `ShowCreateTableBaseSuite` for V2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979740143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r759919791
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
Review comment:
yea I think so
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34719:
URL: https://github.com/apache/spark/pull/34719#issuecomment-979844618
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/145652/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758151668
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableParserSuite.scala
##########
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.parser.CatalystSqlParser.parsePlan
+import org.apache.spark.sql.catalyst.plans.logical.ShowCreateTable
+
+class ShowCreateTableParserSuite extends AnalysisTest {
+ test("SHOW CREATE table") {
Review comment:
```suggestion
test("SHOW CREATE TABLE") {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758183803
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v1/ShowCreateTableSuite.scala
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v1
+
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.execution.command
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that checks V1
+ * table catalogs. The tests that cannot run for all V1 catalogs are located in more
+ * specific test suites:
+ *
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+ * `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends command.ShowCreateTableSuiteBase
+ with command.TestsV1AndV2Commands {
+
+ test("CATS") {
+ val table = "ddl_test"
+ withTable(table) {
+ sql(
+ s"""CREATE TABLE $table
+ |USING json
+ |PARTITIONED BY (c)
+ |CLUSTERED BY (a) SORTED BY (b) INTO 2 BUCKETS
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('a' = '1')
+ |AS SELECT 1 AS a, "foo" AS b, 2.5 AS c
+ """.stripMargin
+ )
+ checkCreateTable(table)
+ }
+ }
+
+ protected def checkCreateTable(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "TABLE", serde)
+ }
+
+ protected def checkCreateView(table: String, serde: Boolean = false): Unit = {
+ checkCreateTableOrView(TableIdentifier(table, Some("default")), "VIEW", serde)
+ }
+
+ protected def checkCreateTableOrView(
+ table: TableIdentifier,
+ checkType: String,
+ serde: Boolean): Unit = {
+ val db = table.database.getOrElse("default")
+ val expected = spark.sharedState.externalCatalog.getTable(db, table.table)
+ val shownDDL = if (serde) {
+ sql(s"SHOW CREATE TABLE ${table.quotedString} AS SERDE").head().getString(0)
+ } else {
+ sql(s"SHOW CREATE TABLE ${table.quotedString}").head().getString(0)
+ }
+ sql(s"DROP $checkType ${table.quotedString}")
+ try {
+ sql(shownDDL)
+ val actual = spark.sharedState.externalCatalog.getTable(db, table.table)
+ checkCatalogTables(expected, actual)
+ } finally {
+ sql(s"DROP $checkType IF EXISTS ${table.table}")
+ }
+ }
+
+ protected def checkCatalogTables(expected: CatalogTable, actual: CatalogTable): Unit = {
+ assert(CatalogTable.normalize(actual) == CatalogTable.normalize(expected))
+ }
+}
+
+/**
+ * The class contains tests for the `SHOW CREATE TABLE` command to check V1 In-Memory
+ * table catalog.
+ */
+class ShowCreateTableSuite extends ShowCreateTableSuiteBase with CommandSuiteBase {
+ override def commandVersion: String = super[ShowCreateTableSuiteBase].commandVersion
+
+ test("SHOW CREATE TABLE") {
+ val t = "tbl"
+ withTable(t) {
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint,
+ | c bigint,
+ | `extraCol` ARRAY<INT>,
+ | `<another>` STRUCT<x: INT, y: ARRAY<BOOLEAN>>
+ |)
+ |$defaultUsing
+ |OPTIONS (
+ | from = 0,
+ | to = 1,
+ | via = 2)
+ |COMMENT 'This is a comment'
+ |TBLPROPERTIES ('prop1' = '1', 'prop2' = '2', 'prop3' = 3, 'prop4' = 4)
+ |PARTITIONED BY (a)
+ |LOCATION '/tmp'
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ assert(showDDL === Array(
+ s"CREATE TABLE `default`.`$t` (",
+ "`b` BIGINT,",
+ "`c` BIGINT,",
+ "`extraCol` ARRAY<INT>,",
+ "`<another>` STRUCT<`x`: INT, `y`: ARRAY<BOOLEAN>>,",
+ "`a` BIGINT NOT NULL)",
+ defaultUsing,
+ "OPTIONS (",
+ "`from` '0',",
+ "`to` '1',",
+ "`via` '2')",
Review comment:
Yeah. I think our SQL parser support both syntaxes.
I write a test case for test it
```scala
test("test") {
val t = "testcat.ns1.ns2.tbl"
withTable(t) {
sql(
s"""
|CREATE TABLE $t (
| a bigint NOT NULL,
| b bigint
|)
|USING foo
|OPTIONS (
| `from` '0',
| `to` '1',
| `via` '2')
""".stripMargin)
val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
showDDL.foreach { i =>
println(i)
}
}
}
```
The result is
```scala
CREATE TABLE testcat.ns1.ns2.tbl (
`a` BIGINT NOT NULL,
`b` BIGINT)
USING foo
OPTIONS(
'from' = '0',
'to' = '1',
'via' = '2')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758229408
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
Review comment:
No, I will change it that put it into `SQLViewSuite`. It's already failed in the analysis phase before the resolve phase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Peng-Lei commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
Peng-Lei commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r758237125
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowCreateTableSuiteBase.scala
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{AnalysisException, QueryTest}
+import org.apache.spark.sql.sources.SimpleInsertSource
+import org.apache.spark.util.Utils
+
+/**
+ * This base suite contains unified tests for the `SHOW CREATE TABLE` command that check V1 and V2
+ * table catalogs. The tests that cannot run for all supported catalogs are located in more
+ * specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowCreateTableSuite`
+ * - V1 table catalog tests:
+ * `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowCreateTableSuite`
+ * - V1 Hive External catalog:
+* `org.apache.spark.sql.hive.execution.command.ShowCreateTableSuite`
+ */
+trait ShowCreateTableSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW CREATE TABLE"
+
+ test("SPARK-33892: SHOW CREATE TABLE w/ char/varchar") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(s"CREATE TABLE $t(v VARCHAR(3), c CHAR(5)) $defaultUsing")
+ val rest = sql(s"SHOW CREATE TABLE $t").head().getString(0)
+ assert(rest.contains("VARCHAR(3)"))
+ assert(rest.contains("CHAR(5)"))
+ }
+ }
+
+ test("DO NOT SUPPORT TEMP VIEW") {
+ val viewName = "spark_28383"
+ withTempView(viewName) {
+ sql(s"CREATE TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$viewName is a temp view. 'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+
+ withGlobalTempView(viewName) {
+ sql(s"CREATE GLOBAL TEMPORARY VIEW $viewName AS SELECT 1 AS a")
+ val globalTempViewDb = spark.sessionState.catalog.globalTempViewManager.database
+ val ex = intercept[AnalysisException] {
+ sql(s"SHOW CREATE TABLE $globalTempViewDb.$viewName")
+ }
+ assert(ex.getMessage.contains(
+ s"$globalTempViewDb.$viewName is a temp view. " +
+ "'SHOW CREATE TABLE' expects a table or permanent view."))
+ }
+ }
+
+ test("SPARK-36012: ADD NULL FLAG WHEN SHOW CREATE TABLE") {
+ val db = "ns1"
+ val table = "tbl"
+ withNamespaceAndTable(db, table) { t =>
+ sql(
+ s"""
+ |CREATE TABLE $t (
+ | a bigint NOT NULL,
+ | b bigint
+ |)
+ |USING ${classOf[SimpleInsertSource].getName}
+ """.stripMargin)
+ val showDDL = getShowCreateDDL(s"SHOW CREATE TABLE $t")
+ // if v2 showDDL(0) == s"CREATE TABLE $t ("
+ // if v1 showDDL(0) == s"CREATE TABLE `$db`.`$table` ("
Review comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #34719: [SPARK-37381][SQL] Unify v1 and v2 SHOW CREATE TABLE tests
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #34719:
URL: https://github.com/apache/spark/pull/34719#discussion_r785580022
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/SQLViewTestSuite.scala
##########
@@ -591,4 +605,52 @@ class PersistedViewTestSuite extends SQLViewTestSuite with SharedSparkSession {
s" The view ${table.qualifiedName} may have been tampered with"))
}
}
+
+ test("show create table for persisted simple view") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a")
+ val expected = "CREATE VIEW `default`.`v1` ( `a`) AS SELECT 1 AS a"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted view with output columns") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS a, 2 AS b", Seq("a", "b COMMENT 'b column'"))
+ val expected = "CREATE VIEW `default`.`v1` ( `a`, `b` COMMENT 'b column')" +
+ " AS SELECT 1 AS a, 2 AS b"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
+ }
+ }
+ }
+
+ test("show create table for persisted simple view with table comment and properties") {
+ val viewName = "v1"
+ Seq(true, false).foreach { serde =>
+ withView(viewName) {
+ createView(viewName, "SELECT 1 AS c1, '2' AS c2", Seq("c1 COMMENT 'bla'", "c2"),
+ Seq("COMMENT 'table comment'", "TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')"))
+
+ val expected = "CREATE VIEW `default`.`v1` ( `c1` COMMENT 'bla', `c2`)" +
+ " COMMENT 'table comment'" +
+ " TBLPROPERTIES ( 'prop1' = 'value1', 'prop2' = 'value2')" +
+ " AS SELECT 1 AS c1, '2' AS c2"
+ assert(getShowCreateDDL(formattedViewName(viewName), serde) == expected)
Review comment:
looks like we should sort TBLPROPERTIES
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org