You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Amanda Liu (Jira)" <ji...@apache.org> on 2024/12/11 01:14:00 UTC
[jira] [Updated] (SPARK-50541) Describe Table As JSON
[ https://issues.apache.org/jira/browse/SPARK-50541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amanda Liu updated SPARK-50541:
-------------------------------
Description:
Support DESCRIBE TABLE ... [AS JSON] option to display table metadata in JSON format.
*Context:*
The Spark SQL command DESCRIBE TABLE displays table metadata in a DataFrame format geared toward human consumption. This format causes parsing challenges, e.g. if fields contain special characters or the format changes as new features are added.
The new AS JSON option would return the table metadata as a JSON string that supports parsing via machine, while being extensible with a minimized risk of breaking changes. It is not meant to be human-readable.
*SQL Ref Spec:*
{ DESC | DESCRIBE }
[ TABLE ] [ EXTENDED | FORMATTED ] table_name \{ [ PARTITION clause ] | [ column_name ] } [ AS JSON ]
*JSON Schema:*
```
{
"table_name": "<table_name>",
"catalog_name": [...],
"database_name": [...],
"qualified_name": "<qualified_name>"
"type": "<table_type>",
"provider": "<provider>",
"columns": [
{ "id": 1, "name": "<name>", "type": <type_json>, "comment": "<comment>", "default": "<default_val>" }
],
"partition_values":
{ "<col_name>": "<val>" }
,
"location": "<path>",
"view_definition": "<view_defn>",
"owner": "<owner>",
"comment": "<comment>",
"table_properties":
{ "property1": "<property1>", "property2": "<property2>" }
,
"storage_properties":
{ "property1": "<property1>", "property2": "<property2>" }
,
"serde_library": "<serde_library>",
"inputformat": "<input_format>",
"outputformt": "<output_format>",
"bucket_columns": [<col_name>],
"sort_columns": [<col_name>],
"created_time": "<timestamp>",
"last_access": "<timestamp>",
"partition_provider": "<partition_provider>"
}
```
was:
Support DESCRIBE TABLE ... [AS JSON] option to display table metadata in JSON format.
*Context:*
The Spark SQL command DESCRIBE TABLE displays table metadata in a DataFrame format geared toward human consumption. This format causes parsing challenges, e.g. if fields contain special characters or the format changes as new features are added. [DBT|https://www.getdbt.com/] is an example customer that motivates this proposal, as providing a structured JSON format can help prevent breakages in pipelines that depend on parsing table metadata.
The new AS JSON option would return the table metadata as a JSON string that supports parsing via machine, while being extensible with a minimized risk of breaking changes. It is not meant to be human-readable.
*SQL Ref Spec:*
{ DESC | DESCRIBE } [ TABLE ] [ EXTENDED | FORMATTED ] table_name \{ [ PARTITION clause ] | [ column_name ] } [ AS JSON ]
*JSON Schema:*
```
{
"table_name": "<table_name>",
"catalog_name": [...],
"database_name": [...],
"qualified_name": "<qualified_name>"
"type": "<table_type>",
"provider": "<provider>",
"columns": [
{
"id": 1,
"name": "<name>",
"type": <type_json>,
"comment": "<comment>",
"default": "<default_val>"
}
],
"partition_values": {
"<col_name>": "<val>"
},
"location": "<path>",
"view_definition": "<view_defn>",
"owner": "<owner>",
"comment": "<comment>",
"table_properties": {
"property1": "<property1>",
"property2": "<property2>"
},
"storage_properties": {
"property1": "<property1>",
"property2": "<property2>"
},
"serde_library": "<serde_library>",
"inputformat": "<input_format>",
"outputformt": "<output_format>",
"bucket_columns": [<col_name>],
"sort_columns": [<col_name>],
"created_time": "<timestamp>",
"last_access": "<timestamp>",
"partition_provider": "<partition_provider>"
}
```
> Describe Table As JSON
> ----------------------
>
> Key: SPARK-50541
> URL: https://issues.apache.org/jira/browse/SPARK-50541
> Project: Spark
> Issue Type: Task
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Amanda Liu
> Priority: Major
>
> Support DESCRIBE TABLE ... [AS JSON] option to display table metadata in JSON format.
>
> *Context:*
> The Spark SQL command DESCRIBE TABLE displays table metadata in a DataFrame format geared toward human consumption. This format causes parsing challenges, e.g. if fields contain special characters or the format changes as new features are added.
> The new AS JSON option would return the table metadata as a JSON string that supports parsing via machine, while being extensible with a minimized risk of breaking changes. It is not meant to be human-readable.
>
> *SQL Ref Spec:*
> { DESC | DESCRIBE }
> [ TABLE ] [ EXTENDED | FORMATTED ] table_name \{ [ PARTITION clause ] | [ column_name ] } [ AS JSON ]
>
> *JSON Schema:*
> ```
> {
> "table_name": "<table_name>",
> "catalog_name": [...],
> "database_name": [...],
> "qualified_name": "<qualified_name>"
> "type": "<table_type>",
> "provider": "<provider>",
> "columns": [
> { "id": 1, "name": "<name>", "type": <type_json>, "comment": "<comment>", "default": "<default_val>" }
> ],
> "partition_values":
> { "<col_name>": "<val>" }
> ,
> "location": "<path>",
> "view_definition": "<view_defn>",
> "owner": "<owner>",
> "comment": "<comment>",
> "table_properties":
> { "property1": "<property1>", "property2": "<property2>" }
> ,
> "storage_properties":
> { "property1": "<property1>", "property2": "<property2>" }
> ,
> "serde_library": "<serde_library>",
> "inputformat": "<input_format>",
> "outputformt": "<output_format>",
> "bucket_columns": [<col_name>],
> "sort_columns": [<col_name>],
> "created_time": "<timestamp>",
> "last_access": "<timestamp>",
> "partition_provider": "<partition_provider>"
> }
> ```
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org