You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2026/07/22 10:47:18 UTC
[PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
pan3793 opened a new pull request, #57429:
URL: https://github.com/apache/spark/pull/57429
### What changes were proposed in this pull request?
Add a new config `spark.sql.hive.showCreateTableAsSerde` (default `false`). When enabled,
plain `SHOW CREATE TABLE` on a Hive serde table routes to the existing
`ShowCreateTableAsSerdeCommand` and renders Hive DDL, instead of converting to Spark DDL.
### Why are the changes needed?
SPARK-27946 (#24938) changed `SHOW CREATE TABLE` to always emit Spark DDL, which fails for
Hive tables whose serde/fileformat has no Spark data source mapping (e.g., RCFILE) and for
transactional Hive tables. The current behavior also confuses users that `CREATE TABLE ... STORED AS Parquet` but `SHOW CREATE TABLE ...` displays `USING Parquet`.
The config restores the pre- SPARK-27946 Hive DDL output for users who need the original behavior.
### Does this PR introduce _any_ user-facing change?
Yes. New config `spark.sql.hive.showCreateTableAsSerde`, default `false` (no behavior
change). When `true`, `SHOW CREATE TABLE` on Hive serde tables returns Hive DDL; data
source tables and views are unaffected.
### How was this patch tested?
Added tests to `o.a.s.sql.hive.execution.command.ShowCreateTableSuite` covering: Hive
table (matches `AS SERDE` output), data source table and view (unaffected), RCFILE and
transactional Hive tables (Hive DDL rendered instead of erroring).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: KIMI K3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5047493824
CI failure is unrelated, will be fixed by https://github.com/apache/spark/pull/57431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5060228580
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/0790f4d6a4cc2fce2f3a84e78b43f41856796289
- merged into branch-4.x https://github.com/apache/spark/commit/eb8956e5ac6aa2ee9d9c30632062352a4dd3550d
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5053393754
thank you, @viirya, I updated the PR description to mention the background.
cc @cloud-fan @gengliangwang, are you fine with adding this config?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5053403612
Rebased to fix the compile error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #57429:
URL: https://github.com/apache/spark/pull/57429#discussion_r3630510680
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -455,7 +431,9 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
isOverwrite,
partition)
- case ShowCreateTable(ResolvedV1TableOrViewIdentifier(ident), asSerde, output) if asSerde =>
+ case ShowCreateTable(ResolvedV1TableOrViewIdentifier(ident), asSerde, output) if asSerde ||
+ (conf.hiveTableShowCreateTableAsSerde &&
+ DDLUtils.isHiveTable(catalogManager.v1SessionCatalog.getTableRawMetadata(ident))) =>
Review Comment:
Good point, done. The guard now matches `ResolvedTable(catalog, _, t: V1Table, _)` directly and reuses `t.catalogTable` instead of re-fetching via `getTableRawMetadata(ident)` — same pattern as the existing `SHOW CREATE TABLE` case just below.
The view handling is split into its own case since `ResolvedV1TableOrViewIdentifier` covered both: views keep the explicit `AS SERDE` routing, and do not need the config check because `isHiveTable` is always false for them (`provider` is `None` — temp views are built without one, and `HiveExternalCatalog.restoreTableMetadata` skips `restoreHiveSerdeTable` for view-like tables). Views also never go through the Hive-to-Spark DDL conversion this config bypasses, so plain `SHOW CREATE TABLE` on a view is unchanged either way.
Existing `ShowCreateTableSuite` (core v1/v2 + Hive) and view suites (`PersistedViewTestSuite`, temp view suites, `HiveSQLViewSuite`) all pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5045652954
cc @viirya @ulysses-you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #57429: [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables
URL: https://github.com/apache/spark/pull/57429
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57429:
URL: https://github.com/apache/spark/pull/57429#issuecomment-5060233156
thank you for the review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-58260][SQL] Add config to render Hive DDL in SHOW CREATE TABLE for Hive tables [spark]
Posted by "ulysses-you (via GitHub)" <gi...@apache.org>.
ulysses-you commented on code in PR #57429:
URL: https://github.com/apache/spark/pull/57429#discussion_r3630304570
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -455,7 +431,9 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
isOverwrite,
partition)
- case ShowCreateTable(ResolvedV1TableOrViewIdentifier(ident), asSerde, output) if asSerde =>
+ case ShowCreateTable(ResolvedV1TableOrViewIdentifier(ident), asSerde, output) if asSerde ||
+ (conf.hiveTableShowCreateTableAsSerde &&
+ DDLUtils.isHiveTable(catalogManager.v1SessionCatalog.getTableRawMetadata(ident))) =>
Review Comment:
Can we reuse the metadata already in the resolved plan instead of re-fetching it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org