You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/27 08:09:56 UTC
[GitHub] [spark] MaxGekk opened a new pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
MaxGekk opened a new pull request #30937:
URL: https://github.com/apache/spark/pull/30937
### What changes were proposed in this pull request?
1. Port DS V2 tests from `DataSourceV2SQLSuite` to the base test suite `ShowNamespacesSuiteBase` to run those tests for v1 catalogs.
2. Port DS v1 tests from `DDLSuite` to `ShowNamespacesSuiteBase` to run the tests for v2 catalogs too.
### Why are the changes needed?
To improve test coverage.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
By running new test suites:
```
$ build/sbt -Phive-2.3 -Phive-thriftserver "test:testOnly *ShowNamespacesSuite"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551527438
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
Review comment:
- V2 is case sensitive, and does not respect the SQL config `spark.sql.caseSensitive`. It always shows namespaces in **upper case**.
- V1 In-Memory takes into account `spark.sql.caseSensitive` but **when the config is `false`, it shows all names in lower case**. For example, if we create a namespace `AAA`, it shows `aaa` (but this could be behavior of `CREATE NAMESPACE`)
- V1 Hive External catalog doesn't respect `spark.sql.caseSensitive`, and **always lower case namespaces**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754444562
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133624/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753477250
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38181/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753505053
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133592/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753480201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38181/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754397800
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38213/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751442094
**[Test build #133405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133405/testReport)** for PR 30937 at commit [`ad20040`](https://github.com/apache/spark/commit/ad20040bd86fb55130b830cedb1b68a9e772b5e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753468473
**[Test build #133592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133592/testReport)** for PR 30937 at commit [`9e88e0a`](https://github.com/apache/spark/commit/9e88e0a40e0ceabc5eab77188a255b8e1e66e0f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754444562
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133624/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #30937:
URL: https://github.com/apache/spark/pull/30937
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754374973
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38213/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753468473
**[Test build #133592 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133592/testReport)** for PR 30937 at commit [`9e88e0a`](https://github.com/apache/spark/commit/9e88e0a40e0ceabc5eab77188a255b8e1e66e0f9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754459009
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751511310
**[Test build #133409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133409/testReport)** for PR 30937 at commit [`798510e`](https://github.com/apache/spark/commit/798510e70ce39861073d518df49711d4fcd93527).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551753374
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
+ Seq(true, false).foreach { caseSensitive =>
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive.toString) {
+ withNamespace(s"$catalog.AAA", s"$catalog.bbb") {
+ sql(s"CREATE NAMESPACE $catalog.AAA")
+ sql(s"CREATE NAMESPACE $catalog.bbb")
+ runShowNamespacesSql(
+ s"SHOW NAMESPACES IN $catalog",
+ Seq("AAA", "bbb") ++ builtinTopNamespaces)
+ runShowNamespacesSql(s"SHOW NAMESPACES IN $catalog LIKE 'AAA'", Seq("AAA"))
+ runShowNamespacesSql(s"SHOW NAMESPACES IN $catalog LIKE 'aaa'", Seq("AAA"))
Review comment:
Actually, there are 2 things here:
1. How we apply pattern. It is always case insensitive, see https://github.com/apache/spark/blob/3a299aa6480ac22501512cd0310d31a441d7dfdc/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala#L85
2. How we show results or create namespace. For example, if we create `AAA` namespace
- v2 shows it as `AAA` independently from `spark.sql.caseSensitive`
- v1 Hive shows it as `aaa` independently from `spark.sql.caseSensitive`
- v1 in-memory shows it as `AAA` if `spark.sql.caseSensitive = true` and `aaa` if `spark.sql.caseSensitive = false`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751514769
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133409/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751446831
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751442094
**[Test build #133405 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133405/testReport)** for PR 30937 at commit [`ad20040`](https://github.com/apache/spark/commit/ad20040bd86fb55130b830cedb1b68a9e772b5e2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551144001
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowNamespacesSuiteBase.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructType}
+
+/**
+ * This base suite contains unified tests for the `SHOW NAMESPACES` and `SHOW DATABASES` commands
+ * that check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowNamespacesSuite`
+ * - V1 table catalog tests: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuite`
+ * - V1 Hive External catalog: `org.apache.spark.sql.hive.execution.command.ShowNamespacesSuite`
+ */
+trait ShowNamespacesSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW NAMESPACES"
+
+ protected def runShowNamespacesSql(sqlText: String, expected: Seq[String]): Unit = {
+ val df = spark.sql(sqlText)
+ assert(df.schema === new StructType().add("namespace", StringType, false))
+ checkAnswer(df, expected.map(Row(_)))
+ }
+
+ protected def topNamespaces(ns: Seq[String]): Seq[String]
+
+ test("default namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES", topNamespaces(Seq.empty))
Review comment:
how about
```
protected val builtinNamespaces: Seq[String] = Nil
...
Seq.empty ++ builtinNamespaces
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753636297
@cloud-fan @HyukjinKwon Could you review this PR, please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754385182
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38213/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551527438
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
Review comment:
1. V2 is case sensitive, and does not respect the SQL config `spark.sql.caseSensitive`. It always shows namespaces in **upper case**.
2. V1 In-Memory takes into account `spark.sql.caseSensitive` but **when the config is `false`, it shows all names in lower case**. For example, if we create a namespace `AAA`, it shows `aaa` (but this could be behavior of `CREATE NAMESPACE`)
3. V1 Hive External catalog doesn't respect `spark.sql.caseSensitive`, and **always lower case namespaces**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751483625
**[Test build #133409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133409/testReport)** for PR 30937 at commit [`798510e`](https://github.com/apache/spark/commit/798510e70ce39861073d518df49711d4fcd93527).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751491891
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38000/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551176327
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
Review comment:
yea we should fail. We can do it later.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751514769
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133409/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751491894
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38000/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753505053
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133592/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753502055
**[Test build #133592 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133592/testReport)** for PR 30937 at commit [`9e88e0a`](https://github.com/apache/spark/commit/9e88e0a40e0ceabc5eab77188a255b8e1e66e0f9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class ShowNamespacesParserSuite extends AnalysisTest with SharedSparkSession `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751446830
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551167244
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
Review comment:
v1 fails at least https://github.com/apache/spark/pull/30937/files#diff-8cbd4478c9a858099aef4d5eea6fa916e7a6a47efd3e00701c4606e7d1674982R37 . If we want to have the same behavior as v1, we should fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754319939
**[Test build #133624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133624/testReport)** for PR 30937 at commit [`8f8751d`](https://github.com/apache/spark/commit/8f8751dadc2394205ca351cf541aa92bd4a1f2b1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754319939
**[Test build #133624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133624/testReport)** for PR 30937 at commit [`8f8751d`](https://github.com/apache/spark/commit/8f8751dadc2394205ca351cf541aa92bd4a1f2b1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751491894
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38000/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551144655
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
Review comment:
should it fail?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551713850
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
+ Seq(true, false).foreach { caseSensitive =>
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> caseSensitive.toString) {
+ withNamespace(s"$catalog.AAA", s"$catalog.bbb") {
+ sql(s"CREATE NAMESPACE $catalog.AAA")
+ sql(s"CREATE NAMESPACE $catalog.bbb")
+ runShowNamespacesSql(
+ s"SHOW NAMESPACES IN $catalog",
+ Seq("AAA", "bbb") ++ builtinTopNamespaces)
+ runShowNamespacesSql(s"SHOW NAMESPACES IN $catalog LIKE 'AAA'", Seq("AAA"))
+ runShowNamespacesSql(s"SHOW NAMESPACES IN $catalog LIKE 'aaa'", Seq("AAA"))
Review comment:
do you mean V2 is always case insensitive?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751488701
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38000/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551176466
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
Review comment:
can you highlight the differences?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754397800
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38213/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551166586
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
Review comment:
This PR already have case sensitivity tests for v1 In-Memory and Hive catalogs. I had to put the test to catalog specific test suites because the behavior is different in all catalogs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753480201
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38181/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551176098
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowNamespacesSuiteBase.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructType}
+
+/**
+ * This base suite contains unified tests for the `SHOW NAMESPACES` and `SHOW DATABASES` commands
+ * that check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowNamespacesSuite`
+ * - V1 table catalog tests: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuite`
+ * - V1 Hive External catalog: `org.apache.spark.sql.hive.execution.command.ShowNamespacesSuite`
+ */
+trait ShowNamespacesSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW NAMESPACES"
+
+ protected def runShowNamespacesSql(sqlText: String, expected: Seq[String]): Unit = {
+ val df = spark.sql(sqlText)
+ assert(df.schema === new StructType().add("namespace", StringType, false))
+ checkAnswer(df, expected.map(Row(_)))
+ }
+
+ protected def topNamespaces(ns: Seq[String]): Seq[String]
+
+ test("default namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES", topNamespaces(Seq.empty))
Review comment:
or `builtinTopNamespaces`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] MaxGekk commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
MaxGekk commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551167728
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowNamespacesSuiteBase.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructType}
+
+/**
+ * This base suite contains unified tests for the `SHOW NAMESPACES` and `SHOW DATABASES` commands
+ * that check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowNamespacesSuite`
+ * - V1 table catalog tests: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuite`
+ * - V1 Hive External catalog: `org.apache.spark.sql.hive.execution.command.ShowNamespacesSuite`
+ */
+trait ShowNamespacesSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW NAMESPACES"
+
+ protected def runShowNamespacesSql(sqlText: String, expected: Seq[String]): Unit = {
+ val df = spark.sql(sqlText)
+ assert(df.schema === new StructType().add("namespace", StringType, false))
+ checkAnswer(df, expected.map(Row(_)))
+ }
+
+ protected def topNamespaces(ns: Seq[String]): Seq[String]
+
+ test("default namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES", topNamespaces(Seq.empty))
Review comment:
I added `topNamespaces()` because built-in namespace `default` exists only on top level.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-753473579
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38181/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751483625
**[Test build #133409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133409/testReport)** for PR 30937 at commit [`798510e`](https://github.com/apache/spark/commit/798510e70ce39861073d518df49711d4fcd93527).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [WIP][SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-751446405
**[Test build #133405 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133405/testReport)** for PR 30937 at commit [`ad20040`](https://github.com/apache/spark/commit/ad20040bd86fb55130b830cedb1b68a9e772b5e2).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30937:
URL: https://github.com/apache/spark/pull/30937#issuecomment-754426012
**[Test build #133624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133624/testReport)** for PR 30937 at commit [`8f8751d`](https://github.com/apache/spark/commit/8f8751dadc2394205ca351cf541aa92bd4a1f2b1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551143272
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/ShowNamespacesSuiteBase.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{StringType, StructType}
+
+/**
+ * This base suite contains unified tests for the `SHOW NAMESPACES` and `SHOW DATABASES` commands
+ * that check V1 and V2 table catalogs. The tests that cannot run for all supported catalogs are
+ * located in more specific test suites:
+ *
+ * - V2 table catalog tests: `org.apache.spark.sql.execution.command.v2.ShowNamespacesSuite`
+ * - V1 table catalog tests: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuiteBase`
+ * - V1 In-Memory catalog: `org.apache.spark.sql.execution.command.v1.ShowNamespacesSuite`
+ * - V1 Hive External catalog: `org.apache.spark.sql.hive.execution.command.ShowNamespacesSuite`
+ */
+trait ShowNamespacesSuiteBase extends QueryTest with DDLCommandTestUtils {
+ override val command = "SHOW NAMESPACES"
+
+ protected def runShowNamespacesSql(sqlText: String, expected: Seq[String]): Unit = {
+ val df = spark.sql(sqlText)
+ assert(df.schema === new StructType().add("namespace", StringType, false))
+ checkAnswer(df, expected.map(Row(_)))
+ }
+
+ protected def topNamespaces(ns: Seq[String]): Seq[String]
+
+ test("default namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key -> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES", topNamespaces(Seq.empty))
Review comment:
`topNamespaces` -> `defaultNamespace`? We don't force catalog implementations to report the top namespace as the default namespace.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #30937: [SPARK-33919][SQL][TESTS] Unify v1 and v2 SHOW NAMESPACES tests
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #30937:
URL: https://github.com/apache/spark/pull/30937#discussion_r551144416
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/ShowNamespacesSuite.scala
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.command.v2
+
+import org.apache.spark.SparkConf
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.connector.BasicInMemoryTableCatalog
+import org.apache.spark.sql.execution.command
+import org.apache.spark.sql.internal.SQLConf
+
+/**
+ * The class contains tests for the `SHOW NAMESPACES` command to check V2 table catalogs.
+ */
+class ShowNamespacesSuite extends command.ShowNamespacesSuiteBase with CommandSuiteBase {
+ override protected def topNamespaces(ns: Seq[String]): Seq[String] = ns
+
+ override def sparkConf: SparkConf = super.sparkConf
+ .set("spark.sql.catalog.testcat_no_namespace", classOf[BasicInMemoryTableCatalog].getName)
+
+ test("IN namespace doesn't exist") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> catalog) {
+ runShowNamespacesSql("SHOW NAMESPACES in dummy", Seq.empty)
+ }
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1", Seq.empty)
+ runShowNamespacesSql(s"SHOW NAMESPACES in $catalog.ns1.ns3", Seq.empty)
+ }
+
+ test("default v2 catalog doesn't support namespace") {
+ withSQLConf(SQLConf.DEFAULT_CATALOG.key-> "testcat_no_namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+ }
+
+ test("v2 catalog doesn't support namespace") {
+ val errMsg = intercept[AnalysisException] {
+ sql("SHOW NAMESPACES in testcat_no_namespace")
+ }.getMessage
+ assert(errMsg.contains("does not support namespaces"))
+ }
+
+ test("case sensitivity") {
Review comment:
shall we test it with v1 catalog as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org