You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/04/23 18:24:44 UTC
(spark) branch master updated: [SPARK-39660][SQL] Support v2 DESCRIBE TABLE .. PARTITION
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7dc8912211e5 [SPARK-39660][SQL] Support v2 DESCRIBE TABLE .. PARTITION
7dc8912211e5 is described below
commit 7dc8912211e5a233b6ede184123f8893ca156b74
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Thu Apr 23 11:24:31 2026 -0700
[SPARK-39660][SQL] Support v2 DESCRIBE TABLE .. PARTITION
### What changes were proposed in this pull request?
Enable DESCRIBE [EXTENDED] TABLE <v2_table> PARTITION (col=val, ...) for V2 tables that implement SupportsPartitionManagement, bringing parity with the V1/Hive behavior.
Previously any partition spec on a V2 table threw unconditionally: "DESCRIBE does not support partition for v2 tables."
Design: split the logical plan into two, mirroring the ShowTables / ShowTablePartition precedent:
- DescribeRelation — pure table describe (no partition spec), extends UnaryCommand.
- DescribeTablePartition (new) — table + required partitionSpec: PartitionSpec + isExtended, extends V2PartitionCommand.
This means:
- Parser (SparkSqlParser): branches on ctx.partitionSpec to emit either DescribeRelation or DescribeTablePartition.
- CheckAnalysis: no special-casing needed — DescribeTablePartition is naturally caught by the existing V2PartitionCommand case, which validates SupportsPartitionManagement and non-empty partition schema.
- ResolveSessionCatalog: two explicit cases, both producing DescribeTableCommand for V1 tables — no silent _ => Map.empty fallthrough.
- DataSourceV2Strategy: DescribeRelation → DescribeTableExec (unchanged pre-PR signature); DescribeTablePartition → DescribeTablePartitionExec (new).
- DescribeTablePartitionExec (new): validates partition existence via listPartitionIdentifiers (matching ShowTablePartitionExec), delegates base rows (schema + partitioning + clustering) to DescribeTableExec.addBaseDescription, and for EXTENDED appends a # Detailed Partition Information section with partition values rendered using ToPrettyString + escapePathName (consistent with ShowTablePartitionExec).
### Why are the changes needed?
DESCRIBE TABLE .. PARTITION was the only partition command that unconditionally rejected V2 tables, even when the table implements SupportsPartitionManagement. Other partition commands (SHOW TABLE EXTENDED PARTITION, ALTER TABLE .. ADD/DROP PARTITION, etc.) already work through the V2PartitionCommand + ResolvePartitionSpec pipeline. This PR closes that gap using the same infrastructure.
### Does this PR introduce _any_ user-facing change?
Yes. `DESCRIBE TABLE .. PARTITION` supports V2 table after this PR.
### How was this patch tested?
Replaced the single "not supported" test in v2/DescribeTableSuite with eight targeted tests: existing partition (happy path), extended partition info, non-existent partition error, capability error for tables without partition management, partial partition spec, case-insensitive column name, unknown partition column, multi-column partition, and NULL partition value. Also updated DescribeTableParserSuite and PlanResolutionSuite for the new DescribeTablePartition plan. All 34 V2 + 66 V1 [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Sonnet 4.6
Closes #55064 from viirya/SPARK-39660-describe-table-partition-v2.
Authored-by: Liang-Chi Hsieh <vi...@gmail.com>
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
.../src/main/resources/error/error-conditions.json | 5 -
.../sql/catalyst/plans/logical/v2Commands.scala | 14 ++-
.../spark/sql/errors/QueryCompilationErrors.scala | 6 -
.../sql/connect/pipelines/PipelinesHandler.scala | 3 +-
.../catalyst/analysis/ResolveSessionCatalog.scala | 10 +-
.../spark/sql/execution/SparkSqlParser.scala | 15 ++-
.../datasources/v2/DataSourceV2Strategy.scala | 9 +-
.../datasources/v2/DescribeTableExec.scala | 11 +-
.../v2/DescribeTablePartitionExec.scala | 89 ++++++++++++++
.../sql-tests/analyzer-results/describe.sql.out | 2 +-
.../org/apache/spark/sql/SQLQueryTestHelper.scala | 1 +
.../command/DescribeTableParserSuite.scala | 22 +++-
.../execution/command/PlanResolutionSuite.scala | 16 +--
.../execution/command/v2/DescribeTableSuite.scala | 135 +++++++++++++++++++--
.../sql/hive/execution/HiveComparisonTest.scala | 1 +
15 files changed, 289 insertions(+), 50 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json b/common/utils/src/main/resources/error/error-conditions.json
index e13e8104dd53..e6c786640e09 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -8979,11 +8979,6 @@
"Cannot delete from table <table> where <filters>."
]
},
- "_LEGACY_ERROR_TEMP_1111" : {
- "message" : [
- "DESCRIBE does not support partition for v2 tables."
- ]
- },
"_LEGACY_ERROR_TEMP_1114" : {
"message" : [
"The streaming sources in a query do not have a common supported execution mode.",
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
index b857a360544e..f63de8d1e465 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala
@@ -910,7 +910,6 @@ case class SetNamespaceLocation(
*/
case class DescribeRelation(
relation: LogicalPlan,
- partitionSpec: TablePartitionSpec,
isExtended: Boolean,
override val output: Seq[Attribute] = DescribeRelation.getOutputAttrs) extends UnaryCommand {
override def child: LogicalPlan = relation
@@ -922,6 +921,19 @@ object DescribeRelation {
def getOutputAttrs: Seq[Attribute] = DescribeCommandSchema.describeTableAttributes()
}
+/**
+ * The logical plan of the DESCRIBE relation_name PARTITION command.
+ */
+case class DescribeTablePartition(
+ table: LogicalPlan,
+ partitionSpec: PartitionSpec,
+ isExtended: Boolean,
+ override val output: Seq[Attribute] = DescribeRelation.getOutputAttrs)
+ extends V2PartitionCommand {
+ override protected def withNewChildInternal(newChild: LogicalPlan): DescribeTablePartition =
+ copy(table = newChild)
+}
+
/**
* The logical plan of the DESCRIBE relation_name col_name command.
*/
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
index 446c6f2a4784..bd1e876c9fbd 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryCompilationErrors.scala
@@ -1568,12 +1568,6 @@ private[sql] object QueryCompilationErrors extends QueryErrorsBase with Compilat
"filters" -> filters.mkString("[", ", ", "]")))
}
- def describeDoesNotSupportPartitionForV2TablesError(): Throwable = {
- new AnalysisException(
- errorClass = "_LEGACY_ERROR_TEMP_1111",
- messageParameters = Map.empty)
- }
-
def cannotReplaceMissingTableError(
tableIdentifier: Identifier): Throwable = {
new CannotReplaceMissingTableException(tableIdentifier)
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
index 62f060014117..dd4799765040 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/pipelines/PipelinesHandler.scala
@@ -28,7 +28,7 @@ import org.apache.spark.connect.proto.{ExecutePlanResponse, PipelineCommandResul
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace, CreateTable, CreateTableAsSelect, CreateView, DescribeRelation, DropView, InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable, ShowFunctions, ShowTableProperties, ShowTables, ShowViews}
+import org.apache.spark.sql.catalyst.plans.logical.{Command, CreateNamespace, CreateTable, CreateTableAsSelect, CreateView, DescribeRelation, DescribeTablePartition, DropView, InsertIntoStatement, LogicalPlan, RenameTable, ShowColumns, ShowCreateTable, ShowFunctions, ShowTableProperties, ShowTables, ShowViews}
import org.apache.spark.sql.connect.common.DataTypeProtoConverter
import org.apache.spark.sql.connect.service.SessionHolder
import org.apache.spark.sql.execution.command.{ShowCatalogsCommand, ShowNamespacesCommand}
@@ -140,6 +140,7 @@ private[connect] object PipelinesHandler extends Logging {
def blockUnsupportedSqlCommand(queryPlan: LogicalPlan): Unit = {
val allowlistedCommands = Set(
classOf[DescribeRelation],
+ classOf[DescribeTablePartition],
classOf[ShowTables],
classOf[ShowTableProperties],
classOf[ShowNamespacesCommand],
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
index d7d943a2eedb..e2bfaef1e700 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala
@@ -197,10 +197,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
// Use v1 command to describe (temp) view, as v2 catalog doesn't support view yet.
case DescribeRelation(
resolvedChild @ ResolvedV1TableOrViewIdentifier(ident),
- partitionSpec,
isExtended,
output) =>
- DescribeTableCommand(resolvedChild, ident, partitionSpec, isExtended, output)
+ DescribeTableCommand(resolvedChild, ident, Map.empty, isExtended, output)
+
+ case DescribeTablePartition(
+ resolvedChild @ ResolvedV1TableOrViewIdentifier(ident),
+ UnresolvedPartitionSpec(spec, _),
+ isExtended,
+ output) =>
+ DescribeTableCommand(resolvedChild, ident, spec, isExtended, output)
case DescribeColumn(
ResolvedViewIdentifier(ident), column: UnresolvedAttribute, isExtended, output) =>
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
index abc62241a006..b910a3dd6d8a 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala
@@ -30,7 +30,7 @@ import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.{CurrentNamespace,
GlobalTempView, LocalTempView, PersistedView,
PlanWithUnresolvedIdentifier, SchemaEvolution, SchemaTypeEvolution, UnresolvedAttribute,
- UnresolvedIdentifier, UnresolvedNamespace, UnresolvedProcedure}
+ UnresolvedIdentifier, UnresolvedNamespace, UnresolvedPartitionSpec, UnresolvedProcedure}
import org.apache.spark.sql.catalyst.catalog._
import org.apache.spark.sql.catalyst.expressions.{Expression, Literal}
import org.apache.spark.sql.catalyst.parser._
@@ -1440,8 +1440,8 @@ class SparkSqlAstBuilder extends AstBuilder {
}
/**
- * Create a [[DescribeColumn]] or [[DescribeRelation]] or [[DescribeRelationAsJsonCommand]]
- * command.
+ * Create a [[DescribeColumn]], [[DescribeRelation]], [[DescribeTablePartition]], or
+ * [[DescribeRelationJsonCommand]] command.
*/
override def visitDescribeRelation(ctx: DescribeRelationContext): LogicalPlan = withOrigin(ctx) {
val isExtended = ctx.EXTENDED != null || ctx.FORMATTED != null
@@ -1463,7 +1463,7 @@ class SparkSqlAstBuilder extends AstBuilder {
isExtended)
}
} else {
- val partitionSpec = if (ctx.partitionSpec != null) {
+ val rawSpec = if (ctx.partitionSpec != null) {
// According to the syntax, visitPartitionSpec returns `Map[String, Option[String]]`.
visitPartitionSpec(ctx.partitionSpec).map {
case (key, Some(value)) => key -> value
@@ -1474,9 +1474,12 @@ class SparkSqlAstBuilder extends AstBuilder {
Map.empty[String, String]
}
if (asJson) {
- DescribeRelationJsonCommand(relation, partitionSpec, isExtended)
+ // DescribeRelationJsonCommand uses the raw Map directly (V1 path only).
+ DescribeRelationJsonCommand(relation, rawSpec, isExtended)
+ } else if (rawSpec.nonEmpty) {
+ DescribeTablePartition(relation, UnresolvedPartitionSpec(rawSpec), isExtended)
} else {
- DescribeRelation(relation, partitionSpec, isExtended)
+ DescribeRelation(relation, isExtended)
}
}
}
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index 66ce4a637d51..6730673cab02 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -425,12 +425,13 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
case DescribeNamespace(ResolvedNamespace(catalog, ns, _), extended, output) =>
DescribeNamespaceExec(output, catalog.asNamespaceCatalog, ns, extended) :: Nil
- case DescribeRelation(r: ResolvedTable, partitionSpec, isExtended, output) =>
- if (partitionSpec.nonEmpty) {
- throw QueryCompilationErrors.describeDoesNotSupportPartitionForV2TablesError()
- }
+ case DescribeRelation(r: ResolvedTable, isExtended, output) =>
DescribeTableExec(output, r.table, isExtended) :: Nil
+ case DescribeTablePartition(r: ResolvedTable, part, isExtended, output) =>
+ DescribeTablePartitionExec(output, r.table.asPartitionable, r.identifier,
+ Seq(part).asResolvedPartitionSpecs.head, isExtended) :: Nil
+
case DescribeColumn(r: ResolvedTable, column, isExtended, output) =>
column match {
case c: Attribute =>
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
index 3cc767380d8b..51f906724b98 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTableExec.scala
@@ -37,9 +37,7 @@ case class DescribeTableExec(
isExtended: Boolean) extends LeafV2CommandExec {
override protected def run(): Seq[InternalRow] = {
val rows = new ArrayBuffer[InternalRow]()
- addSchema(rows)
- addPartitioning(rows)
- addClustering(rows)
+ addBaseDescription(rows)
if (isExtended) {
addMetadataColumns(rows)
@@ -50,6 +48,13 @@ case class DescribeTableExec(
rows.toSeq
}
+ /** Schema + partitioning + clustering rows, shared with DescribeTablePartitionExec. */
+ private[v2] def addBaseDescription(rows: ArrayBuffer[InternalRow]): Unit = {
+ addSchema(rows)
+ addPartitioning(rows)
+ addClustering(rows)
+ }
+
private def addTableDetails(rows: ArrayBuffer[InternalRow]): Unit = {
rows += emptyRow()
rows += toCatalystRow("# Detailed Table Information", "", "")
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTablePartitionExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTablePartitionExec.scala
new file mode 100644
index 000000000000..6a2c3c441f8f
--- /dev/null
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DescribeTablePartitionExec.scala
@@ -0,0 +1,89 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.collection.mutable.ArrayBuffer
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.ResolvedPartitionSpec
+import org.apache.spark.sql.catalyst.catalog.ExternalCatalogUtils.escapePathName
+import org.apache.spark.sql.catalyst.expressions.{Attribute, Literal, ToPrettyString}
+import org.apache.spark.sql.connector.catalog.{Identifier, SupportsPartitionManagement}
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+case class DescribeTablePartitionExec(
+ output: Seq[Attribute],
+ table: SupportsPartitionManagement,
+ tableIdent: Identifier,
+ partSpec: ResolvedPartitionSpec,
+ isExtended: Boolean) extends LeafV2CommandExec {
+
+ override protected def run(): Seq[InternalRow] = {
+ val partitionRow = validateAndGetPartition()
+
+ // Delegate schema + partitioning + clustering to DescribeTableExec.
+ val rows = new ArrayBuffer[InternalRow]()
+ DescribeTableExec(output, table, isExtended = false).addBaseDescription(rows)
+
+ if (isExtended) {
+ addPartitionDetails(rows, partitionRow)
+ }
+ rows.toSeq
+ }
+
+ private def validateAndGetPartition(): InternalRow = {
+ val partitionSchema = table.partitionSchema()
+ val (names, ident) = (partSpec.names, partSpec.ident)
+ val partitionIdentifiers = table.listPartitionIdentifiers(names.toArray, ident)
+ if (partitionIdentifiers.isEmpty) {
+ throw QueryCompilationErrors.notExistPartitionError(tableIdent, ident, partitionSchema)
+ }
+ assert(partitionIdentifiers.length == 1)
+ partitionIdentifiers.head
+ }
+
+ private def addPartitionDetails(
+ rows: ArrayBuffer[InternalRow],
+ partitionRow: InternalRow): Unit = {
+ val partitionSchema = table.partitionSchema()
+
+ // Render partition values using ToPrettyString + escapePathName,
+ // consistent with ShowTablePartitionExec.
+ val len = partitionSchema.length
+ val partitions = new Array[String](len)
+ val timeZoneId = conf.sessionLocalTimeZone
+ for (i <- 0 until len) {
+ val dataType = partitionSchema(i).dataType
+ val partValueUTF8String = ToPrettyString(Literal(partitionRow.get(i, dataType), dataType),
+ Some(timeZoneId)).eval(null)
+ val partValueStr = if (partValueUTF8String == null) "null" else partValueUTF8String.toString
+ partitions(i) = escapePathName(partitionSchema(i).name) + "=" + escapePathName(partValueStr)
+ }
+ val partitionValues = partitions.mkString("[", ", ", "]")
+
+ rows += toCatalystRow("", "", "")
+ rows += toCatalystRow("# Detailed Partition Information", "", "")
+ rows += toCatalystRow("Partition Values", partitionValues, "")
+
+ val metadata = table.loadPartitionMetadata(partSpec.ident)
+ metadata.asScala.toSeq.sortBy(_._1).foreach { case (k, v) =>
+ rows += toCatalystRow(k, v, "")
+ }
+ }
+}
diff --git a/sql/core/src/test/resources/sql-tests/analyzer-results/describe.sql.out b/sql/core/src/test/resources/sql-tests/analyzer-results/describe.sql.out
index 891d84c0e409..f284b4cabdb4 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/describe.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/describe.sql.out
@@ -379,7 +379,7 @@ ExplainCommand 'DescribeColumn 'b, false, [info_name#x, info_value#x], SimpleMod
-- !query
EXPLAIN DESCRIBE t PARTITION (c='Us', d=2)
-- !query analysis
-ExplainCommand 'DescribeRelation [c=Us, d=2], false, [col_name#x, data_type#x, comment#x], SimpleMode
+ExplainCommand 'DescribeTablePartition unresolvedpartitionspec((c,Us), (d,2), None), false, [col_name#x, data_type#x, comment#x], SimpleMode
-- !query
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala
index 5ad9605c7d78..3cf26aa94a5d 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestHelper.scala
@@ -137,6 +137,7 @@ trait SQLQueryTestHelper extends SQLConfHelper with Logging {
case _: DescribeCommandBase
| _: DescribeColumnCommand
| _: DescribeRelation
+ | _: DescribeTablePartition
| _: DescribeColumn => true
case PhysicalOperation(_, _, Sort(_, true, _, _)) => true
case _ => plan.children.iterator.exists(isSemanticallySorted)
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala
index f8174d24c949..436fa2e2389a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DescribeTableParserSuite.scala
@@ -18,8 +18,10 @@
package org.apache.spark.sql.execution.command
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedAttribute, UnresolvedTableOrView}
-import org.apache.spark.sql.catalyst.plans.logical.{DescribeColumn, DescribeRelation}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisTest, UnresolvedAttribute,
+ UnresolvedPartitionSpec, UnresolvedTableOrView}
+import org.apache.spark.sql.catalyst.plans.logical.{DescribeColumn, DescribeRelation,
+ DescribeTablePartition}
import org.apache.spark.sql.test.SharedSparkSession
class DescribeTableParserSuite extends SharedSparkSession with AnalysisTest {
@@ -28,16 +30,24 @@ class DescribeTableParserSuite extends SharedSparkSession with AnalysisTest {
test("SPARK-17328: Fix NPE with EXPLAIN DESCRIBE TABLE") {
comparePlans(parsePlan("describe t"),
DescribeRelation(
- UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), Map.empty, isExtended = false))
+ UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = false))
comparePlans(parsePlan("describe table t"),
DescribeRelation(
- UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), Map.empty, isExtended = false))
+ UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = false))
comparePlans(parsePlan("describe table extended t"),
DescribeRelation(
- UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), Map.empty, isExtended = true))
+ UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = true))
comparePlans(parsePlan("describe table formatted t"),
DescribeRelation(
- UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), Map.empty, isExtended = true))
+ UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true), isExtended = true))
+ }
+
+ test("describe table with partition spec") {
+ comparePlans(parsePlan("DESCRIBE TABLE t PARTITION (ds='2024-01-01')"),
+ DescribeTablePartition(
+ UnresolvedTableOrView(Seq("t"), "DESCRIBE TABLE", true),
+ UnresolvedPartitionSpec(Map("ds" -> "2024-01-01")),
+ isExtended = false))
}
test("describe table column") {
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
index 92d56e800c72..89fb6eca223e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/PlanResolutionSuite.scala
@@ -27,12 +27,12 @@ import org.mockito.invocation.InvocationOnMock
import org.apache.spark.SparkUnsupportedOperationException
import org.apache.spark.sql.{AnalysisException, SaveMode}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, AsOfVersion, EmptyFunctionRegistry, NoSuchTableException, RelationResolution, ResolvedFieldName, ResolvedFieldPosition, ResolvedIdentifier, ResolvedTable, ResolveSessionCatalog, TimeTravelSpec, UnresolvedAttribute, UnresolvedFieldPosition, UnresolvedInlineTable, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, AnalysisTest, Analyzer, AsOfVersion, EmptyFunctionRegistry, NoSuchTableException, RelationResolution, ResolvedFieldName, ResolvedFieldPosition, ResolvedIdentifier, ResolvedTable, ResolveSessionCatalog, TimeTravelSpec, UnresolvedAttribute, UnresolvedFieldPosition, UnresolvedInlineTable, UnresolvedPartitionSpec, UnresolvedRelation, UnresolvedSubqueryColumnAliases, UnresolvedTable}
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat, CatalogTable, CatalogTableType, InMemoryCatalog, SessionCatalog, TempVariableManager}
import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Cast, EqualTo, Expression, InSubquery, IntegerLiteral, ListQuery, Literal, StringLiteral}
import org.apache.spark.sql.catalyst.expressions.objects.StaticInvoke
import org.apache.spark.sql.catalyst.parser.ParseException
-import org.apache.spark.sql.catalyst.plans.logical.{AlterColumns, AlterColumnSpec, AnalysisOnlyCommand, AppendData, Assignment, CreateTable, CreateTableAsSelect, DefaultValueExpression, DeleteAction, DeleteFromTable, DescribeRelation, DropTable, InsertAction, InsertIntoStatement, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, OverwriteByExpression, OverwritePartitionsDynamic, Project, SetTableLocation, SetTableProperties, ShowTableProperties, SubqueryAlias, UnsetTablePropert [...]
+import org.apache.spark.sql.catalyst.plans.logical.{AlterColumns, AlterColumnSpec, AnalysisOnlyCommand, AppendData, Assignment, CreateTable, CreateTableAsSelect, DefaultValueExpression, DeleteAction, DeleteFromTable, DescribeRelation, DescribeTablePartition, DropTable, InsertAction, InsertIntoStatement, LocalRelation, LogicalPlan, MergeIntoTable, OneRowRelation, OverwriteByExpression, OverwritePartitionsDynamic, Project, SetTableLocation, SetTableProperties, ShowTableProperties, Subquery [...]
import org.apache.spark.sql.catalyst.rules.Rule
import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLId
import org.apache.spark.sql.connector.FakeV2Provider
@@ -943,13 +943,13 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest {
comparePlans(parsed2, expected2)
} else {
parsed1 match {
- case DescribeRelation(_: ResolvedTable, _, isExtended, _) =>
+ case DescribeRelation(_: ResolvedTable, isExtended, _) =>
assert(!isExtended)
case _ => fail("Expect DescribeTable, but got:\n" + parsed1.treeString)
}
parsed2 match {
- case DescribeRelation(_: ResolvedTable, _, isExtended, _) =>
+ case DescribeRelation(_: ResolvedTable, isExtended, _) =>
assert(isExtended)
case _ => fail("Expect DescribeTable, but got:\n" + parsed2.treeString)
}
@@ -965,10 +965,10 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest {
comparePlans(parsed3, expected3)
} else {
parsed3 match {
- case DescribeRelation(_: ResolvedTable, partitionSpec, isExtended, _) =>
+ case DescribeTablePartition(_: ResolvedTable, partitionSpec, isExtended, _) =>
assert(!isExtended)
- assert(partitionSpec == Map("a" -> "1"))
- case _ => fail("Expect DescribeTable, but got:\n" + parsed2.treeString)
+ assert(partitionSpec == UnresolvedPartitionSpec(Map("a" -> "1")))
+ case _ => fail("Expect DescribeTablePartition, but got:\n" + parsed3.treeString)
}
}
}
@@ -1662,7 +1662,7 @@ class PlanResolutionSuite extends SharedSparkSession with AnalysisTest {
case AppendData(r: DataSourceV2Relation, _, _, _, _, _, _) =>
assert(r.catalog.contains(catalog))
assert(r.identifier.exists(_.name() == tableIdent))
- case DescribeRelation(r: ResolvedTable, _, _, _) =>
+ case DescribeRelation(r: ResolvedTable, _, _) =>
assert(r.catalog == catalog)
assert(r.identifier.name() == tableIdent)
case ShowTableProperties(r: ResolvedTable, _, _) =>
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
index 847a956c11ab..1bd6df5cf928 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/v2/DescribeTableSuite.scala
@@ -31,14 +31,135 @@ class DescribeTableSuite extends command.DescribeTableSuiteBase
with CommandSuiteBase {
import testImplicits._
- test("Describing a partition is not supported") {
+ test("DESCRIBE TABLE PARTITION on an existing partition") {
withNamespaceAndTable("ns", "table") { tbl =>
- spark.sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing " +
- "PARTITIONED BY (id)")
- val e = intercept[AnalysisException] {
- sql(s"DESCRIBE TABLE $tbl PARTITION (id = 1)")
- }
- assert(e.message === "DESCRIBE does not support partition for v2 tables.")
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = 1)")
+ // Non-extended describe with a valid partition spec shows schema + partition section,
+ // same output as DESCRIBE TABLE (partition spec is used only for validation).
+ checkAnswer(
+ sql(s"DESCRIBE TABLE $tbl PARTITION (id = 1)"),
+ Seq(
+ Row("id", "bigint", null),
+ Row("data", "string", null),
+ Row("# Partition Information", "", ""),
+ Row("# col_name", "data_type", "comment"),
+ Row("id", "bigint", null)))
+ }
+ }
+
+ test("DESCRIBE TABLE EXTENDED PARTITION shows detailed partition information") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = 1)")
+ val result = sql(s"DESCRIBE TABLE EXTENDED $tbl PARTITION (id = 1)").collect()
+ val colNames = result.map(_.getString(0))
+ // Must include the detailed partition section header
+ assert(colNames.contains("# Detailed Partition Information"))
+ // Must include the partition values row
+ assert(result.exists(r =>
+ r.getString(0) == "Partition Values" && r.getString(1) == "[id=1]"))
+ }
+ }
+
+ test("DESCRIBE TABLE PARTITION on a non-existent partition raises an error") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"DESCRIBE TABLE $tbl PARTITION (id = 999)").collect()
+ },
+ condition = "PARTITIONS_NOT_FOUND",
+ parameters = Map(
+ "partitionList" -> "PARTITION (`id` = 999)",
+ "tableName" -> "`ns`.`table`"))
+ }
+ }
+
+ test("DESCRIBE TABLE PARTITION on a table without partition management raises an error") {
+ withNamespaceAndTable("ns", "table", nonPartitionCatalog) { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing")
+ val sqlText = s"DESCRIBE TABLE $tbl PARTITION (id = 1)"
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(sqlText)
+ },
+ condition = "INVALID_PARTITION_OPERATION.PARTITION_MANAGEMENT_IS_UNSUPPORTED",
+ parameters = Map("name" -> s"`$nonPartitionCatalog`.`ns`.`table`"),
+ queryContext = Array(ExpectedContext(
+ fragment = tbl,
+ start = sqlText.indexOf(tbl),
+ stop = sqlText.indexOf(tbl) + tbl.length - 1)))
+ }
+ }
+
+ test("DESCRIBE TABLE PARTITION with partial partition spec raises an error") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, city string, data string) " +
+ s"$defaultUsing PARTITIONED BY (id, city)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = 1, city = 'NYC')")
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"DESCRIBE TABLE $tbl PARTITION (id = 1)")
+ },
+ condition = "_LEGACY_ERROR_TEMP_1232",
+ parameters = Map(
+ "specKeys" -> "id",
+ "partitionColumnNames" -> "id, city",
+ "tableName" -> s"`$catalog`.`ns`.`table`"))
+ }
+ }
+
+ test("DESCRIBE TABLE PARTITION with case-insensitive partition column name") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = 1)")
+ // Case-insensitive partition column name should normalize.
+ checkAnswer(
+ sql(s"DESCRIBE TABLE $tbl PARTITION (ID = 1)"),
+ Seq(
+ Row("id", "bigint", null),
+ Row("data", "string", null),
+ Row("# Partition Information", "", ""),
+ Row("# col_name", "data_type", "comment"),
+ Row("id", "bigint", null)))
+ }
+ }
+
+ test("DESCRIBE TABLE PARTITION with unknown partition column raises an error") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ checkError(
+ exception = intercept[AnalysisException] {
+ sql(s"DESCRIBE TABLE $tbl PARTITION (xyz = 1)")
+ },
+ condition = "PARTITIONS_NOT_FOUND",
+ parameters = Map(
+ "partitionList" -> "`xyz`",
+ "tableName" -> s"`$catalog`.`ns`.`table`"))
+ }
+ }
+
+ test("DESCRIBE TABLE EXTENDED PARTITION with multi-column partition") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (data string, id bigint, city string) " +
+ s"$defaultUsing PARTITIONED BY (id, city)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = 1, city = 'NYC')")
+ val result = sql(s"DESCRIBE TABLE EXTENDED $tbl PARTITION (id = 1, city = 'NYC')").collect()
+ assert(result.exists(r =>
+ r.getString(0) == "Partition Values" && r.getString(1) == "[id=1, city=NYC]"))
+ }
+ }
+
+ test("DESCRIBE TABLE EXTENDED PARTITION with NULL partition value") {
+ withNamespaceAndTable("ns", "table") { tbl =>
+ sql(s"CREATE TABLE $tbl (id bigint, data string) $defaultUsing PARTITIONED BY (id)")
+ sql(s"ALTER TABLE $tbl ADD PARTITION (id = null)")
+ val result = sql(s"DESCRIBE TABLE EXTENDED $tbl PARTITION (id = null)").collect()
+ val partRow = result.find(_.getString(0) == "Partition Values")
+ assert(partRow.isDefined, s"Partition Values row not found in: ${result.mkString("\n")}")
+ assert(partRow.get.getString(1) == "[id=NULL]",
+ s"Unexpected partition value: ${partRow.get.getString(1)}")
}
}
diff --git a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
index 1f52e25a730b..bbb7d9ea5b76 100644
--- a/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
+++ b/sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveComparisonTest.scala
@@ -375,6 +375,7 @@ abstract class HiveComparisonTest extends SparkFunSuite {
(!hiveQuery.logical.isInstanceOf[DescribeFunction]) &&
(!hiveQuery.logical.isInstanceOf[DescribeCommandBase]) &&
(!hiveQuery.logical.isInstanceOf[DescribeRelation]) &&
+ (!hiveQuery.logical.isInstanceOf[DescribeTablePartition]) &&
(!hiveQuery.logical.isInstanceOf[DescribeColumn]) &&
preparedHive != catalyst) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org