You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by liancheng <gi...@git.apache.org> on 2016/05/02 17:47:49 UTC
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
GitHub user liancheng opened a pull request:
https://github.com/apache/spark/pull/12844
[SPARK-14127][SQL] Native "DESC [EXTENDED | FORMATTED] <table>" DDL command
## What changes were proposed in this pull request?
This PR implements native `DESC [EXTENDED | FORMATTED] <table>` DDL command. Sample output:
```
scala> spark.sql("desc extended src").show(100, truncate = false)
+----------------------------+---------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+---------------------------------+-------+
|key |int | |
|value |string | |
| | | |
|# Detailed Table Information|CatalogTable(`default`.`src`, ...| |
+----------------------------+---------------------------------+-------+
scala> spark.sql("desc formatted src").show(100, truncate = false)
+----------------------------+----------------------------------------------------------+-------+
|col_name |data_type |comment|
+----------------------------+----------------------------------------------------------+-------+
|key |int | |
|value |string | |
| | | |
|# Detailed Table Information| | |
|Database: |default | |
|Owner: |lian | |
|Create Time: |Mon Jan 04 17:06:00 CST 2016 | |
|Last Access Time: |Thu Jan 01 08:00:00 CST 1970 | |
|Location: |hdfs://localhost:9000/user/hive/warehouse_hive121/src | |
|Table Type: |MANAGED | |
|Table Parameters: | | |
| transient_lastDdlTime |1451898360 | |
| | | |
|# Storage Information | | |
|SerDe Library: |org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe | |
|InputFormat: |org.apache.hadoop.mapred.TextInputFormat | |
|OutputFormat: |org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat| |
|Num Buckets: |-1 | |
|Bucket Columns: |[] | |
|Sort Columns: |[] | |
|Storage Desc Parameters: | | |
| serialization.format |1 | |
+----------------------------+----------------------------------------------------------+-------+
```
## How was this patch tested?
A test case is added to `HiveDDLSuite` to check command output.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/liancheng/spark spark-14127-desc-table
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12844.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12844
----
commit 89e0aea04bdeffaa69be0c19f10860d12af90953
Author: Cheng Lian <li...@databricks.com>
Date: 2016-05-02T17:07:32Z
Implements native 'DESC TABE' DDL command
commit 718da25b489ef56e2b1092484bd6fbcdf2a547ed
Author: Cheng Lian <li...@databricks.com>
Date: 2016-05-02T17:39:24Z
Test case
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61955420
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala ---
@@ -48,6 +48,7 @@ case class CatalogStorageFormat(
inputFormat: Option[String],
outputFormat: Option[String],
serde: Option[String],
+ compressed: Boolean,
--- End diff --
Is this ever true? If it isn't we could leave it out.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61956502
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,88 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ buffer += Row("# Partition Information", "", "")
+ buffer += Row(s"# ${output(0).name}", output(1).name, output(2).name)
+
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", relation.catalogTable.toString, "")
+ }
+
+ private def describeFormatted(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ val table = relation.catalogTable
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", "", "")
+ buffer += Row("Database:", table.database, "")
+ buffer += Row("Owner:", table.owner, "")
+ buffer += Row("Create Time:", new Date(table.createTime).toString, "")
+ buffer += Row("Last Access Time:", new Date(table.lastAccessTime).toString, "")
+ buffer += Row("Location:", table.storage.locationUri.getOrElse(""), "")
+ buffer += Row("Table Type:", table.tableType.name, "")
+
+ buffer += Row("Table Parameters:", "", "")
+ table.properties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+
+ buffer += Row("", "", "")
+ buffer += Row("# Storage Information", "", "")
+ table.storage.serde.foreach(serdeLib => buffer += Row("SerDe Library:", serdeLib, ""))
+ table.storage.inputFormat.foreach(format => buffer += Row("InputFormat:", format, ""))
+ table.storage.outputFormat.foreach(format => buffer += Row("OutputFormat:", format, ""))
+ buffer += Row("Compressed:", if (table.storage.compressed) "Yes" else "No", "")
+ buffer += Row("Num Buckets:", table.numBuckets.toString, "")
+ buffer += Row("Bucket Columns:", table.bucketColumnNames.mkString("[", ", ", "]"), "")
+ buffer += Row("Sort Columns:", table.sortColumnNames.mkString("[", ", ", "]"), "")
+
+ buffer += Row("Storage Desc Parameters:", "", "")
+ table.storage.serdeProperties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+ }
+
+ private def describeSchema(schema: StructType, buffer: ArrayBuffer[Row]): Unit = {
+ schema.foreach { column =>
+ val comment =
+ if (column.metadata.contains("comment")) column.metadata.getString("comment") else ""
--- End diff --
Just an idea: So 9/10 times I use `column.metadata` it is to get the comment. Shouldn't we just add this to the StructField...?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216500747
**[Test build #57630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57630/consoleFull)** for PR 12844 at commit [`18b9bb5`](https://github.com/apache/spark/commit/18b9bb50039d75150dca06b4f97bcffbe41343e7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216779054
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61848409
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1012,7 +1017,7 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
entry("field.delim", ctx.fieldsTerminatedBy) ++
entry("serialization.format", ctx.fieldsTerminatedBy) ++
entry("escape.delim", ctx.escapedBy) ++
- entry("colelction.delim", ctx.collectionItemsTerminatedBy) ++
--- End diff --
A typo bug, not related to this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216759826
**[Test build #57729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57729/consoleFull)** for PR 12844 at commit [`b5dbc15`](https://github.com/apache/spark/commit/b5dbc155afc447a0f700919f7645fc093ec26925).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216442310
**[Test build #57597 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57597/consoleFull)** for PR 12844 at commit [`803f28e`](https://github.com/apache/spark/commit/803f28ed39c5013dfe63cfef404e78317cc1f980).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216444192
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57601/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216474567
**[Test build #57613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57613/consoleFull)** for PR 12844 at commit [`9194fe1`](https://github.com/apache/spark/commit/9194fe1112ed247ff10d37ee6864ef88a9bd07fb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216474675
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216779058
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57729/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216309281
**[Test build #57539 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57539/consoleFull)** for PR 12844 at commit [`718da25`](https://github.com/apache/spark/commit/718da25b489ef56e2b1092484bd6fbcdf2a547ed).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216442313
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57597/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216327624
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216442022
**[Test build #57597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57597/consoleFull)** for PR 12844 at commit [`803f28e`](https://github.com/apache/spark/commit/803f28ed39c5013dfe63cfef404e78317cc1f980).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216443382
**[Test build #57601 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57601/consoleFull)** for PR 12844 at commit [`0bc9f5a`](https://github.com/apache/spark/commit/0bc9f5a7a604f2f10015ddef5049c692d6d97041).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216464548
**[Test build #57613 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57613/consoleFull)** for PR 12844 at commit [`9194fe1`](https://github.com/apache/spark/commit/9194fe1112ed247ff10d37ee6864ef88a9bd07fb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216493503
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216518030
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216474678
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57613/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61956298
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,88 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ buffer += Row("# Partition Information", "", "")
+ buffer += Row(s"# ${output(0).name}", output(1).name, output(2).name)
+
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", relation.catalogTable.toString, "")
+ }
+
+ private def describeFormatted(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
--- End diff --
We could also use some sort of an `append(string, string, string)` function instead of calling `buffer += Row(..., ..., ...)` alot. But i guess that is just a matter of preference.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61955524
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala ---
@@ -48,6 +48,7 @@ case class CatalogStorageFormat(
inputFormat: Option[String],
outputFormat: Option[String],
serde: Option[String],
+ compressed: Boolean,
--- End diff --
Nvm. Hive can pass compressed tables.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216667089
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61992394
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,88 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ buffer += Row("# Partition Information", "", "")
+ buffer += Row(s"# ${output(0).name}", output(1).name, output(2).name)
+
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", relation.catalogTable.toString, "")
+ }
+
+ private def describeFormatted(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
--- End diff --
Makes sense, actually there were several times that I forgot to add the trailing empty string(s) while working on this PR. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216789517
Thanks for the review! I'm merging this to master and branch-2.0.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216480598
**[Test build #57621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57621/consoleFull)** for PR 12844 at commit [`a66885a`](https://github.com/apache/spark/commit/a66885a545d670aa5ddc94126284446c246ca81c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216493509
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57621/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61956757
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1012,6 +1017,7 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
entry("field.delim", ctx.fieldsTerminatedBy) ++
entry("serialization.format", ctx.fieldsTerminatedBy) ++
entry("escape.delim", ctx.escapedBy) ++
+ // The following typo is inherited from Hive...
--- End diff --
+1 for this comment. Totally got me the last time.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216442312
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216518036
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57630/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r62952863
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,92 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ append(buffer, "# Partition Information", "", "")
+ append(buffer, s"# ${output(0).name}", output(1).name, output(2).name)
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ append(buffer, "", "", "")
+ append(buffer, "# Detailed Table Information", relation.catalogTable.toString, "")
--- End diff --
@liancheng To improve the output of `Explain`, I plan to change the default implementation of `toString` of `case class CatalogTable`. That will also affect the output of `Describe Extended`.
I checked what Hive did for the command `Describe Extended`, as follows.
`Detailed Table Information Table(tableName:t1, dbName:default, owner:root, createTime:1462627092, lastAccessTime:0, retention:0, sd:StorageDescriptor(cols:[FieldSchema(name:col, type:int, comment:null)], location:hdfs://6b68a24121f4:9000/user/hive/warehouse/t1, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{serialization.format=1}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), partitionKeys:[], parameters:{transient_lastDdlTime=1462627092}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)`
Basically, in the implementation of `toString`, I will try to follow what you did in `describeFormatted` but the contents will be in a single line. Feel free to let me know if you have any concern or suggestion. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216327447
**[Test build #57539 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57539/consoleFull)** for PR 12844 at commit [`718da25`](https://github.com/apache/spark/commit/718da25b489ef56e2b1092484bd6fbcdf2a547ed).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12844
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61856248
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkSqlParser.scala ---
@@ -1012,7 +1017,7 @@ class SparkSqlAstBuilder(conf: SQLConf) extends AstBuilder {
entry("field.delim", ctx.fieldsTerminatedBy) ++
entry("serialization.format", ctx.fieldsTerminatedBy) ++
entry("escape.delim", ctx.escapedBy) ++
- entry("colelction.delim", ctx.collectionItemsTerminatedBy) ++
--- End diff --
Oh wait... It's actually a typo from Hive... :dizzy_face:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216517719
**[Test build #57630 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57630/consoleFull)** for PR 12844 at commit [`18b9bb5`](https://github.com/apache/spark/commit/18b9bb50039d75150dca06b4f97bcffbe41343e7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r62055855
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,88 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ buffer += Row("# Partition Information", "", "")
+ buffer += Row(s"# ${output(0).name}", output(1).name, output(2).name)
+
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", relation.catalogTable.toString, "")
+ }
+
+ private def describeFormatted(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ val table = relation.catalogTable
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", "", "")
+ buffer += Row("Database:", table.database, "")
+ buffer += Row("Owner:", table.owner, "")
+ buffer += Row("Create Time:", new Date(table.createTime).toString, "")
+ buffer += Row("Last Access Time:", new Date(table.lastAccessTime).toString, "")
+ buffer += Row("Location:", table.storage.locationUri.getOrElse(""), "")
+ buffer += Row("Table Type:", table.tableType.name, "")
+
+ buffer += Row("Table Parameters:", "", "")
+ table.properties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+
+ buffer += Row("", "", "")
+ buffer += Row("# Storage Information", "", "")
+ table.storage.serde.foreach(serdeLib => buffer += Row("SerDe Library:", serdeLib, ""))
+ table.storage.inputFormat.foreach(format => buffer += Row("InputFormat:", format, ""))
+ table.storage.outputFormat.foreach(format => buffer += Row("OutputFormat:", format, ""))
+ buffer += Row("Compressed:", if (table.storage.compressed) "Yes" else "No", "")
+ buffer += Row("Num Buckets:", table.numBuckets.toString, "")
+ buffer += Row("Bucket Columns:", table.bucketColumnNames.mkString("[", ", ", "]"), "")
+ buffer += Row("Sort Columns:", table.sortColumnNames.mkString("[", ", ", "]"), "")
+
+ buffer += Row("Storage Desc Parameters:", "", "")
+ table.storage.serdeProperties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+ }
+
+ private def describeSchema(schema: StructType, buffer: ArrayBuffer[Row]): Unit = {
+ schema.foreach { column =>
+ val comment =
+ if (column.metadata.contains("comment")) column.metadata.getString("comment") else ""
--- End diff --
I am not saying that we should (re)move it. I am only suggesting that it might be easier if we had such an accessor in StructField.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216778695
**[Test build #57729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57729/consoleFull)** for PR 12844 at commit [`b5dbc15`](https://github.com/apache/spark/commit/b5dbc155afc447a0f700919f7645fc093ec26925).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by liancheng <gi...@git.apache.org>.
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/12844#discussion_r61992398
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala ---
@@ -290,29 +291,88 @@ case class DescribeTableCommand(table: TableIdentifier, isExtended: Boolean)
val result = new ArrayBuffer[Row]
sparkSession.sessionState.catalog.lookupRelation(table) match {
case catalogRelation: CatalogRelation =>
- catalogRelation.catalogTable.schema.foreach { column =>
- result += Row(column.name, column.dataType, column.comment.orNull)
- }
-
- if (catalogRelation.catalogTable.partitionColumns.nonEmpty) {
- result += Row("# Partition Information", "", "")
- result += Row(s"# ${output(0).name}", output(1).name, output(2).name)
-
- catalogRelation.catalogTable.partitionColumns.foreach { col =>
- result += Row(col.name, col.dataType, col.comment.orNull)
- }
+ if (isExtended) {
+ describeExtended(catalogRelation, result)
+ } else if (isFormatted) {
+ describeFormatted(catalogRelation, result)
+ } else {
+ describe(catalogRelation, result)
}
case relation =>
- relation.schema.fields.foreach { field =>
- val comment =
- if (field.metadata.contains("comment")) field.metadata.getString("comment") else ""
- result += Row(field.name, field.dataType.simpleString, comment)
- }
+ describeSchema(relation.schema, result)
}
result
}
+
+ // Shows data columns and partitioned columns (if any)
+ private def describe(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describeSchema(relation.catalogTable.schema, buffer)
+
+ if (relation.catalogTable.partitionColumns.nonEmpty) {
+ buffer += Row("# Partition Information", "", "")
+ buffer += Row(s"# ${output(0).name}", output(1).name, output(2).name)
+
+ describeSchema(relation.catalogTable.partitionColumns, buffer)
+ }
+ }
+
+ private def describeExtended(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", relation.catalogTable.toString, "")
+ }
+
+ private def describeFormatted(relation: CatalogRelation, buffer: ArrayBuffer[Row]): Unit = {
+ describe(relation, buffer)
+
+ val table = relation.catalogTable
+
+ buffer += Row("", "", "")
+ buffer += Row("# Detailed Table Information", "", "")
+ buffer += Row("Database:", table.database, "")
+ buffer += Row("Owner:", table.owner, "")
+ buffer += Row("Create Time:", new Date(table.createTime).toString, "")
+ buffer += Row("Last Access Time:", new Date(table.lastAccessTime).toString, "")
+ buffer += Row("Location:", table.storage.locationUri.getOrElse(""), "")
+ buffer += Row("Table Type:", table.tableType.name, "")
+
+ buffer += Row("Table Parameters:", "", "")
+ table.properties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+
+ buffer += Row("", "", "")
+ buffer += Row("# Storage Information", "", "")
+ table.storage.serde.foreach(serdeLib => buffer += Row("SerDe Library:", serdeLib, ""))
+ table.storage.inputFormat.foreach(format => buffer += Row("InputFormat:", format, ""))
+ table.storage.outputFormat.foreach(format => buffer += Row("OutputFormat:", format, ""))
+ buffer += Row("Compressed:", if (table.storage.compressed) "Yes" else "No", "")
+ buffer += Row("Num Buckets:", table.numBuckets.toString, "")
+ buffer += Row("Bucket Columns:", table.bucketColumnNames.mkString("[", ", ", "]"), "")
+ buffer += Row("Sort Columns:", table.sortColumnNames.mkString("[", ", ", "]"), "")
+
+ buffer += Row("Storage Desc Parameters:", "", "")
+ table.storage.serdeProperties.foreach { case (key, value) =>
+ buffer += Row(s" $key", value, "")
+ }
+ }
+
+ private def describeSchema(schema: StructType, buffer: ArrayBuffer[Row]): Unit = {
+ schema.foreach { column =>
+ val comment =
+ if (column.metadata.contains("comment")) column.metadata.getString("comment") else ""
--- End diff --
I think the metadata is heavily used in ML code. Another thing is that the data type API is already public for a long time. We probably don't want to change it unless there are particular good reasons.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216444182
**[Test build #57601 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57601/consoleFull)** for PR 12844 at commit [`0bc9f5a`](https://github.com/apache/spark/commit/0bc9f5a7a604f2f10015ddef5049c692d6d97041).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216493391
**[Test build #57621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57621/consoleFull)** for PR 12844 at commit [`a66885a`](https://github.com/apache/spark/commit/a66885a545d670aa5ddc94126284446c246ca81c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216444191
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14127][SQL] Native "DESC [EXTENDED | FO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12844#issuecomment-216327626
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57539/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org