You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "viirya (via GitHub)" <gi...@apache.org> on 2026/03/14 23:23:25 UTC
[PR] [][SQL] Support CREATE TABLE LIKE for V2 [spark]
viirya opened a new pull request, #54809:
URL: https://github.com/apache/spark/pull/54809
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
8. If you want to add or modify an error type or message, please read the guideline first in
'common/utils/src/main/resources/error/README.md'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. Documentation-only updates are not considered user-facing changes.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
If benchmark tests were added, please run the benchmarks in GitHub Actions for the consistent environment, and the instructions could accord to: https://spark.apache.org/developer-tools.html#github-workflow-benchmarks.
-->
### Was this patch authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this patch, please include the
phrase: 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2943468801
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -514,6 +514,35 @@ case class CreateTable(
}
}
+/**
+ * Create a new table with the same schema/partitioning as an existing table or view,
+ * for use with a v2 catalog.
+ *
+ * @param name Target table identifier. Starts as UnresolvedIdentifier, resolved to
+ * ResolvedIdentifier by ResolveCatalogs.
+ * @param source Source table or view. Starts as UnresolvedTableOrView, resolved to
+ * ResolvedTable / ResolvedPersistentView / ResolvedTempView by ResolveRelations.
+ * @param fileFormat User-specified STORED AS / ROW FORMAT (Hive-style). Empty if not specified.
+ * @param provider User-specified USING provider. None if not specified.
+ * @param properties User-specified TBLPROPERTIES.
+ * @param ifNotExists IF NOT EXISTS flag.
+ */
+case class CreateTableLike(
Review Comment:
can we have one single command (UnaryRunnableCommand)? I thought that's the preferred way now to reduce plan complexity in the different stages
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #54809: [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2
URL: https://github.com/apache/spark/pull/54809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941406618
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
Review Comment:
Oh, this can be `V1Table` that wraps `CatalogTable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2978672448
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -615,6 +615,35 @@ case class CreateTable(
}
}
+/**
+ * Create a new table with the same schema/partitioning as an existing table or view,
+ * for use with a v2 catalog.
+ *
+ * @param name Target table identifier. Starts as UnresolvedIdentifier, resolved to
+ * ResolvedIdentifier by ResolveCatalogs.
+ * @param source Source table or view. Starts as UnresolvedTableOrView, resolved to
+ * ResolvedTable / ResolvedPersistentView / ResolvedTempView by ResolveRelations.
+ * @param fileFormat User-specified STORED AS / ROW FORMAT (Hive-style). Empty if not specified.
+ * @param provider User-specified USING provider. None if not specified.
+ * @param properties User-specified TBLPROPERTIES.
+ * @param ifNotExists IF NOT EXISTS flag.
+ */
+case class CreateTableLike(
+ name: LogicalPlan,
+ source: LogicalPlan,
+ fileFormat: CatalogStorageFormat,
Review Comment:
Replacing with just `location: Option[String]` would indeed lose the serde info. But the plan could carry `SerdeInfo` (which is already a catalyst class at `sql/catalyst/.../plans/logical/statements.scala`) instead of `CatalogStorageFormat`. `SerdeInfo` captures the user's intent (`storedAs = "parquet"`, custom serde class, serde properties) without pulling in the resolved Hive I/O classes.
`ResolveSessionCatalog` can then reconstruct the full `CatalogStorageFormat` before passing it to `CreateTableLikeCommand` — the `inputFormat`/`outputFormat`/`serde` are all derived from the `STORED AS` format name via `HiveSerDe.sourceToSerDe()` (a static lookup). This keeps the V2 logical plan free of V1 catalog concepts while preserving full backward compatibility for the V1 fallback path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2978676198
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -615,6 +615,35 @@ case class CreateTable(
}
}
+/**
+ * Create a new table with the same schema/partitioning as an existing table or view,
+ * for use with a v2 catalog.
+ *
+ * @param name Target table identifier. Starts as UnresolvedIdentifier, resolved to
+ * ResolvedIdentifier by ResolveCatalogs.
+ * @param source Source table or view. Starts as UnresolvedTableOrView, resolved to
+ * ResolvedTable / ResolvedPersistentView / ResolvedTempView by ResolveRelations.
+ * @param fileFormat User-specified STORED AS / ROW FORMAT (Hive-style). Empty if not specified.
+ * @param provider User-specified USING provider. None if not specified.
+ * @param properties User-specified TBLPROPERTIES.
+ * @param ifNotExists IF NOT EXISTS flag.
+ */
+case class CreateTableLike(
+ name: LogicalPlan,
+ source: LogicalPlan,
+ fileFormat: CatalogStorageFormat,
Review Comment:
```
case class CreateTableLike(
name: LogicalPlan,
source: LogicalPlan,
location: Option[String],
provider: Option[String],
serdeInfo: Option[SerdeInfo], // already exists in Spark — storedAs, serde, formatClasses, props
properties: Map[String, String],
ifNotExists: Boolean)
```
Then in ResolveSessionCatalog, reconstruct the CatalogStorageFormat before passing to CreateTableLikeCommand:
```
case CreateTableLike(ResolvedV1Identifier(targetIdent),
ResolvedV1TableOrViewIdentifier(sourceIdent),
location, provider, serdeInfo, properties, ifNotExists) =>
val storage = toStorageFormat(location, serdeInfo, ctx) // same helper the parser uses
CreateTableLikeCommand(targetIdent, sourceIdent, storage, provider, properties, ifNotExists)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964782450
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,36 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause) — it does NOT contain properties copied from the
+ * source table. Connectors that want to copy source-format-specific metadata (e.g. Delta
+ * protocol version, Iceberg sort order, format version) should read it directly from
+ * {@code sourceTable}.
+ * <p>
+ * The default implementation falls back to {@link #createTable(Identifier, TableInfo)}, which
+ * creates a new table using only the schema and partitioning extracted from {@code sourceTable}
+ * by Spark, plus the user-specified overrides in {@code tableInfo}. Connectors with
+ * format-specific copy semantics should override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides (TBLPROPERTIES, LOCATION), resolved provider, and
+ * current user as owner; source TBLPROPERTIES are NOT bulk-copied
+ * @param sourceTable the resolved source table whose metadata is being copied
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ return createTable(ident, tableInfo);
Review Comment:
Is this actually safe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4073615362
@aokolnychyi Yeah, that's exactly I'm thinking about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4074691213
The downside of adding a new method like `createTableLike` is that the eventual behavior becomes connector-based, meaning Spark can't fully control it. The question is whether we can populate enough information via the public properties to make this command usable for Delta and Iceberg. For instance, Iceberg needs to pick the format version and Delta needs to pick the protocol version (inherit it). One way to expose that is via table properties.
@viirya, can you do a deeper analysis to ensure we pick the right design but also make it usable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941422395
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider and location are added if determined above.
+ val locationProp: Option[(String, String)] =
+ fileFormat.locationUri.map(uri =>
+ TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp
+
+ try {
+ // Constraints from the source table are intentionally NOT copied for several reasons:
Review Comment:
This comment is too long to be included here, let's shorten it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964799256
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
This seems inconsistent. It does not seem like TableInfo contains only overrides, it seems like it actually copies some state from the source table. And if it copies partitioning, why doesn't it copy constraints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2966796298
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,36 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause) — it does NOT contain properties copied from the
+ * source table. Connectors that want to copy source-format-specific metadata (e.g. Delta
+ * protocol version, Iceberg sort order, format version) should read it directly from
+ * {@code sourceTable}.
+ * <p>
+ * The default implementation falls back to {@link #createTable(Identifier, TableInfo)}, which
+ * creates a new table using only the schema and partitioning extracted from {@code sourceTable}
+ * by Spark, plus the user-specified overrides in {@code tableInfo}. Connectors with
+ * format-specific copy semantics should override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides (TBLPROPERTIES, LOCATION), resolved provider, and
+ * current user as owner; source TBLPROPERTIES are NOT bulk-copied
+ * @param sourceTable the resolved source table whose metadata is being copied
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ return createTable(ident, tableInfo);
Review Comment:
Yea, this is what I wanted to ask. Should we provide at least a default create table like for connectors not implement this API and just do minimum thing like columns and partitioning? But by doing that, we have to carry columns and partitions in the TableInfo so it raises the question in https://github.com/apache/spark/pull/54809#discussion_r2964799256.
A consistent way might be just throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4129275069
Thanks @aokolnychyi @gengliangwang @sarutak @gengliangwang @jzhuge @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941466438
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
Review Comment:
Do we have V1 -> V2 within/across catalog create tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4069276193
BTW, consider unifying to a single CreateTableLikeExec — The current PR keeps two execution paths: V1 fallback via CreateTableLikeCommand (for V1-V1 cases) and the new CreateTableLikeExec (for V2 targets). The test "v2 source, v1 target" already proves CreateTableLikeExec works for session catalog targets via V2SessionCatalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944051690
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider and location are added if determined above.
+ val locationProp: Option[(String, String)] =
+ fileFormat.locationUri.map(uri =>
+ TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp
+
+ try {
+ // Constraints from the source table are intentionally NOT copied for several reasons:
+ // 1. V1 tables (CatalogTable) have no constraint objects — CHECK, PRIMARY KEY,
+ // UNIQUE and FOREIGN KEY are V2-only concepts (introduced in Spark 4.1.0).
+ // CreateTableLikeCommand had nothing to copy, so not copying here preserves
+ // behavioral parity with the V1 path.
+ // 2. ForeignKey constraints carry a refTable Identifier bound to the source catalog;
+ // copying them to a different catalog creates dangling cross-catalog references.
+ // 3. Constraint names must be unique within a namespace; blindly copying them risks
+ // collisions with existing constraints in the target namespace.
+ // 4. Validation status (VALID/INVALID/UNVALIDATED) is tied to the source table's
+ // existing data and is meaningless on a newly created empty target table.
+ // 5. NOT NULL is already captured in Column.nullable() and copied via withColumns.
+ // 6. Consistent with PostgreSQL semantics: CREATE TABLE LIKE does not include
+ // constraints by default; users must explicitly opt in via INCLUDING CONSTRAINTS.
+ // If constraint copying is desired, use ALTER TABLE ADD CONSTRAINT after creation.
+ // If we wanted to support them in the future, the right approach would be to add an
+ // INCLUDING CONSTRAINTS clause (as PostgreSQL does) rather than copying blindly.
+ val tableInfo = new TableInfo.Builder()
Review Comment:
Good point. CatalogV2Util.convertTableProperties (used by CreateTableExec) calls withDefaultOwnership to add the current user as owner. We should do the same by adding CatalogV2Util.withDefaultOwnership(finalProps). I'll add that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944064480
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
Review Comment:
Valid. Only locationUri is used in CreateTableLikeExec. I'll change the exec's parameter to location: Option[URI] and extract it at the DataSourceV2Strategy callsite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4085196246
> How can we address / minimize discrepancies between implementations? For instance, what if one connector preserves constraints while the other drops them? What if one connector copies props from the source while the other drops them? Anything we can do beyond docs?
Documentation helps but doesn't enforce consistency. Here's a more structural idea.
The root cause of discrepancy is that the target connector has to know what is important to copy from the source — but it has no way to know source-format-specific semantics. Delta knows which properties are internal and must be filtered. Iceberg knows its sort order and format version must be preserved. A generic target connector cannot know any of this.
One way to address this is to introduce a source-side contract: instead of the target being responsible for deciding what to copy, the source table exposes a pre-built TableInfo representing what it considers correct to copy for a LIKE operation. Something like a new method on the Table interface:
```
default TableInfo tableInfoForLike() {
// default: schema + partitioning only (current behavior)
}
```
The flow would then be:
1. Spark calls sourceTable.tableInfoForLike() to get source-prepared metadata
2. Spark merges user overrides (TBLPROPERTIES, LOCATION, USING) on top
3. Spark calls targetCatalog.createTableLike(ident, mergedTableInfo, sourceTable)
This cleanly separates responsibilities:
* Source controls what it exposes for copying — Delta filters out delta.columnMapping.maxColumnId, coordinated commits props, etc.; Iceberg includes sort order and format version
* Target applies user overrides and creates the table using its own format semantics
The remaining discrepancy is only on the target side — e.g., a target that doesn't support constraints simply drops them. This is unavoidable and acceptable: you cannot force a target format to support features it doesn't have. But at least the source's intent is clearly expressed and correctly packaged, rather than relying on the target to reverse-engineer it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4085360256
> What would be the meaning of `TableInfo` in this API? Will it only contain overrides or will it contain a merged state? One potential benefit of passing merged state from Spark is to have more consistent behavior. That said, connectors still will have to filter the props separately and it was kind of nice to have overrides explicitly.
I'd recommend overrides only — tableInfo should contain only what the user explicitly specified in the command (TBLPROPERTIES, LOCATION, USING). Pre-merging in Spark would bring back the disambiguation problem that connectors would need to reverse-engineer which properties came from the user vs Spark, losing the clean separation that makes createTableLike better than the capability approach. With overrides-only, the contract is clear: read source state from sourceTable, apply user intent from tableInfo, connector decides what gets copied and how to merge. This matches your earlier intuition that "it was kind of nice to have overrides explicitly."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941393993
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
Review Comment:
Do we have tests for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941399943
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
Review Comment:
Does this mean it would only work for creating V2 table from another V2 table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2976694336
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
+ * It does NOT contain schema, partitioning, properties, or constraints from the source table.
+ * Connectors must read all source metadata directly from {@code sourceTable}, including
+ * columns ({@link Table#columns()}), partitioning ({@link Table#partitioning()}),
+ * constraints ({@link Table#constraints()}), and format-specific properties
+ * ({@link Table#properties()}).
+ * <p>
+ * The default implementation throws {@link UnsupportedOperationException}. Connectors that
+ * support {@code CREATE TABLE ... LIKE ...} must override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides only: TBLPROPERTIES, LOCATION, resolved provider,
+ * and owner; source schema, partitioning, and constraints are NOT included
+ * @param sourceTable the resolved source table; connectors read schema, partitioning,
+ * constraints, properties, and any format-specific metadata from this object
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @throws UnsupportedOperationException If the catalog does not support CREATE TABLE LIKE
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
Review Comment:
1. shall we move `sourceTable` before `tableInfo`?
2. shall we rename tableInfo as `userProvidedTableInfo` or `userSpecifiedTableInfo`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4088557904
I don't see how `tableInfoForLike` described [here](https://github.com/apache/spark/pull/54809#issuecomment-4085196246) helps in our case. It would still mean the implementation is connector-dependent. Also, there would be no way express things like Iceberg sort order because it is not part of Spark `TableInfo`.
I also don't think `createTableLike` with `sourceIdent` is any better. In fact, it would break cross-catalog migration and would move the resolution further into the connector.
All in all, I think `createTableLike(ident, tableInfo, sourceTable)` with `tableInfo` containing overrides is the best approach we considered. Any thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4088353433
I'm +1 for @viirya 's [solution](https://github.com/apache/spark/pull/54809#issuecomment-4085196246) for now.
@gengliangwang
Passing `Identifier sourceIdent` instead of `Table sourceTable` breaks cross-catalog `CREATE TABLE LIKE` right? Consider:
```
-- Iceberg source -> Delta target
CREATE TABLE delta_catalog.db.target LIKE iceberg_catalog.db.source
```
In this solution, `delta_catalog` receives `sourceIdent = db.source`, but it cannot call `loadTable(db.source)` because that identifier belongs to `iceberg_catalog`, not `delta_catalog`. The target catalog simply has no way to resolve an identifier from a different catalog.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4068894871
cc @szehon-ho as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2978140694
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -615,6 +615,35 @@ case class CreateTable(
}
}
+/**
+ * Create a new table with the same schema/partitioning as an existing table or view,
+ * for use with a v2 catalog.
+ *
+ * @param name Target table identifier. Starts as UnresolvedIdentifier, resolved to
+ * ResolvedIdentifier by ResolveCatalogs.
+ * @param source Source table or view. Starts as UnresolvedTableOrView, resolved to
+ * ResolvedTable / ResolvedPersistentView / ResolvedTempView by ResolveRelations.
+ * @param fileFormat User-specified STORED AS / ROW FORMAT (Hive-style). Empty if not specified.
+ * @param provider User-specified USING provider. None if not specified.
+ * @param properties User-specified TBLPROPERTIES.
+ * @param ifNotExists IF NOT EXISTS flag.
+ */
+case class CreateTableLike(
+ name: LogicalPlan,
+ source: LogicalPlan,
+ fileFormat: CatalogStorageFormat,
Review Comment:
A full replacement to location: Option[String] would break the V1 fallback: ResolveSessionCatalog passes the entire CatalogStorageFormat (including serde class, input/output format for Hive-style STORED AS/ROW FORMAT) to CreateTableLikeCommand. There is no way to reconstruct that from just location. Instead, added a Scaladoc comment on the field explaining why it is retained.
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/CreateTableLikeSuite.scala:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, InMemoryCatalog}
+import org.apache.spark.sql.types.{CharType, IntegerType, LongType, StringType, VarcharType}
+
+class CreateTableLikeSuite extends DatasourceV2SQLBase {
+
+ private def testCatalog: InMemoryCatalog =
+ catalog("testcat").asInstanceOf[InMemoryCatalog]
+
+ private def testCatalog2: InMemoryCatalog =
+ catalog("testcat2").asInstanceOf[InMemoryCatalog]
+
+ // -------------------------------------------------------------------------
+ // Basic V2 path
+ // -------------------------------------------------------------------------
+
+ test("v2 target, v1 source: schema and partitioning are copied") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet PARTITIONED BY (data)")
+ sql("CREATE TABLE testcat.dst LIKE src")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ assert(dst.schema() === CatalogV2Util.v2ColumnsToStructType(dst.columns()))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ // partition column encoded in partitioning (identity transform on the partition column)
+ assert(dst.partitioning.nonEmpty)
+ }
+ }
+
+ test("v2 target, v2 source: pure v2 path") {
+ withTable("testcat.src", "testcat.dst") {
+ sql("CREATE TABLE testcat.src (id bigint, data string) USING foo")
+ sql("CREATE TABLE testcat.dst LIKE testcat.src")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+
+ test("cross-catalog: source in testcat, target in testcat2") {
+ withTable("testcat.src", "testcat2.dst") {
+ sql("CREATE TABLE testcat.src (id bigint, name string) USING foo")
+ sql("CREATE TABLE testcat2.dst LIKE testcat.src")
+
+ val dst = testCatalog2.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "name"))
+ }
+ }
+
+ test("3-part name: catalog.namespace.table for both target and source") {
+ withTable("testcat.ns.src", "testcat2.ns.dst") {
+ sql("CREATE NAMESPACE IF NOT EXISTS testcat.ns")
+ sql("CREATE NAMESPACE IF NOT EXISTS testcat2.ns")
+ sql("CREATE TABLE testcat.ns.src (id bigint, data string) USING foo")
+ sql("CREATE TABLE testcat2.ns.dst LIKE testcat.ns.src")
+
+ val dst = testCatalog2.loadTable(Identifier.of(Array("ns"), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // IF NOT EXISTS
+ // -------------------------------------------------------------------------
+
+ test("IF NOT EXISTS: second call is silent when table already exists") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint) USING parquet")
+ sql("CREATE TABLE testcat.dst LIKE src")
+ // Should not throw
+ sql("CREATE TABLE IF NOT EXISTS testcat.dst LIKE src")
+ }
+ }
+
+ test("without IF NOT EXISTS, duplicate create throws TableAlreadyExistsException") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint) USING parquet")
+ sql("CREATE TABLE testcat.dst LIKE src")
+ intercept[TableAlreadyExistsException] {
+ sql("CREATE TABLE testcat.dst LIKE src")
+ }
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Views as source
+ // -------------------------------------------------------------------------
+
+ test("persistent view as source, v2 target") {
+ withTable("src", "testcat.dst") {
+ withView("v") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet")
+ sql("INSERT INTO src VALUES (1, 'a')")
+ sql("CREATE VIEW v AS SELECT * FROM src")
+ sql("CREATE TABLE testcat.dst LIKE v")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+ }
+
+ test("temp view as source, v2 target") {
+ withTable("src", "testcat.dst") {
+ withTempView("tv") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet")
+ sql("CREATE TEMP VIEW tv AS SELECT id, data FROM src")
+ sql("CREATE TABLE testcat.dst LIKE tv")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Property / provider behavior
+ // -------------------------------------------------------------------------
+
+ test("source TBLPROPERTIES are copied to target when connector implements createTableLike") {
+ // InMemoryTableCatalog overrides createTableLike to merge source properties into the target,
+ // demonstrating connector-specific copy semantics. Connectors that do not override
+ // createTableLike fall back to createTable and do not copy source properties.
Review Comment:
Fixed.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
Review Comment:
Updated the Javadoc.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -240,6 +240,20 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
qualifyLocInTableSpec(tableSpec), options, ifNotExists) :: Nil
}
+ // CREATE TABLE ... LIKE ... for a v2 catalog target.
+ // Source is an already-resolved Table object; no extra catalog round-trip is needed.
+ // Views are wrapped in V1Table so the exec can extract schema and provider uniformly.
+ case CreateTableLike(
+ ResolvedIdentifier(catalog, ident), source,
+ fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
+ val table = source match {
+ case ResolvedTable(_, _, t, _) => t
+ case ResolvedPersistentView(_, _, meta) => V1Table(meta)
+ case ResolvedTempView(_, meta) => V1Table(meta)
+ }
+ CreateTableLikeExec(catalog.asTableCatalog, ident, table,
+ fileFormat.locationUri, provider, properties, ifNotExists) :: Nil
Review Comment:
Added the same qualification logic used by qualifyLocInTableSpec.
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
+ * It does NOT contain schema, partitioning, properties, or constraints from the source table.
+ * Connectors must read all source metadata directly from {@code sourceTable}, including
+ * columns ({@link Table#columns()}), partitioning ({@link Table#partitioning()}),
+ * constraints ({@link Table#constraints()}), and format-specific properties
+ * ({@link Table#properties()}).
+ * <p>
+ * The default implementation throws {@link UnsupportedOperationException}. Connectors that
+ * support {@code CREATE TABLE ... LIKE ...} must override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides only: TBLPROPERTIES, LOCATION, resolved provider,
+ * and owner; source schema, partitioning, and constraints are NOT included
+ * @param sourceTable the resolved source table; connectors read schema, partitioning,
+ * constraints, properties, and any format-specific metadata from this object
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @throws UnsupportedOperationException If the catalog does not support CREATE TABLE LIKE
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
Review Comment:
Renamed parameter and reordered signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4072711861
> The proposed behavior seems different from `CREATE TABLE LIKE` in [Databricks Runtime](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-table-like#notes). Is it OK?
>
> I wonder if we can delegate what to copy on `CREATE TABLE LIKE` to each table format implementation?
DBR's behavior is format-aware: what gets copied depends on the source/target format combination. A Delta target gets constraints and configuration; a non-Delta target doesn't. This implies the copy logic is delegated to the table format (Delta) rather than being hardcoded in a single generic exec node.
In our implementation, CreateTableLikeExec is format-agnostic — it always copies only columns, partitioning, and provider, regardless of what the source or target catalog supports.
The "delegate to format implementation" design (where Delta controls what it copies) is appealing but would require a new TableCatalog API (e.g. createTableLike), which is a larger scope. For this PR we kept the behavior simple and consistent with existing V1 CreateTableLikeCommand. I'm happy to file a follow-up JIRA to explore catalog-delegated copy semantics, including comments and constraint propagation for supporting catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4073600207
I took a look at the PR with fresh eyes today. I do think @sarutak brings a valid point.
What about actually offering a new method in `TableCatalog`?
```
default Table createTableLike(
Identifier ident,
TableInfo tableInfo,
Table sourceTable) throws TableAlreadyExistsException, NoSuchNamespaceException {
throws QueryExecutionErrors.unsupportedCreateTableLikeError();
}
```
Where `tableInfo` will contain override / explicit options and connectors can access `sourceTable` for any additional information? It will be a bit harder to standardize what props / state has to be copied. The danger right now is that Spark may copy properties that can't be copied and the connector can't distinguish between CREATE and CREATE LIKE.
This is similar to what Iceberg does with its `snapshot` and `migrate` procedures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4085107033
@jzhuge Thanks for the Iceberg examples — sort order and format-version are exactly the kind of format-specific semantics that can't be handled generically. This further confirms createTableLike is the right direction.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4129137203
Thanks, merging to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941416295
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
Review Comment:
Isn't this source provider but not target? Can we actually populate this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4069186216
The proposed behavior seems different from `CREATE TABLE LIKE` in [Databricks Runtime](https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-create-table-like#notes). Is it OK?
I wonder if we can delegate what to copy on `CREATE TABLE LIKE` to each table format implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2942075282
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider and location are added if determined above.
+ val locationProp: Option[(String, String)] =
+ fileFormat.locationUri.map(uri =>
+ TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp
+
+ try {
+ // Constraints from the source table are intentionally NOT copied for several reasons:
Review Comment:
Okay. I'll shorten it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2942055079
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
Review Comment:
Yes — the test "CHAR and VARCHAR types are preserved from v1 source to v2 target" in CreateTableLikeSuite covers this. It creates a V1 source with CHAR(10) and VARCHAR(20), runs CREATE TABLE testcat.dst LIKE src, and asserts schema("name").dataType === CharType(10) and schema("tag").dataType === VarcharType(20).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4067138227
I'll take a look later today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941735028
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -240,6 +240,33 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
qualifyLocInTableSpec(tableSpec), options, ifNotExists) :: Nil
}
+ // CREATE TABLE ... LIKE ... for a v2 catalog target.
+ // Source is an already-resolved Table object; no extra catalog round-trip is needed.
+ case CreateTableLike(
+ ResolvedIdentifier(catalog, ident),
+ ResolvedTable(_, _, table, _),
+ fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
+ CreateTableLikeExec(
+ catalog.asTableCatalog, ident, table, fileFormat, provider, properties, ifNotExists) :: Nil
Review Comment:
The three `CreateTableLike` match cases (for `ResolvedTable`, `ResolvedPersistentView`, `ResolvedTempView`) are nearly identical. Consider consolidating into a single pattern:
```scala
case CreateTableLike(
ResolvedIdentifier(catalog, ident), source,
fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
val table = source match {
case ResolvedTable(_, _, t, _) => t
case ResolvedPersistentView(_, _, meta) => V1Table(meta)
case ResolvedTempView(_, meta) => V1Table(meta)
}
CreateTableLikeExec(
catalog.asTableCatalog, ident, table, fileFormat, provider, properties, ifNotExists) :: Nil
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
Review Comment:
`fileFormat: CatalogStorageFormat` carries `inputFormat`/`outputFormat`/`serde` fields, but only `locationUri` is used (line 84). Consider narrowing the exec's parameter to `location: Option[URI]` to make the contract explicit, leaving the full `CatalogStorageFormat` only in the logical plan (where the V1 fallback path needs it).
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala:
##########
@@ -721,83 +721,76 @@ class DDLParserSuite extends AnalysisTest with SharedSparkSession {
}
test("create table like") {
- val v1 = "CREATE TABLE table1 LIKE table2"
+ // Helper to extract fields from the new CreateTableLike unresolved plan.
+ // The parser now emits CreateTableLike (v2 logical plan) instead of
+ // CreateTableLikeCommand, so both name and source are unresolved identifiers.
Review Comment:
The source is `UnresolvedTableOrView`, not an unresolved identifier:
```suggestion
// CreateTableLikeCommand, so the name is an UnresolvedIdentifier and the source is an UnresolvedTableOrView.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2942073160
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
Review Comment:
Yes, this is the source provider being copied to the target — which is exactly the semantics of CREATE TABLE LIKE: the target inherits the source's format unless overridden by a USING clause. This matches V1 CreateTableLikeCommand behavior, which also copies the source provider. The copied provider goes into PROP_PROVIDER in finalProps and is passed to catalog.createTable. Whether the target catalog uses it is catalog-specific: InMemoryCatalog stores it as-is; V2SessionCatalog validates it via DataSource.lookupDataSource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944062410
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -240,6 +240,33 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
qualifyLocInTableSpec(tableSpec), options, ifNotExists) :: Nil
}
+ // CREATE TABLE ... LIKE ... for a v2 catalog target.
+ // Source is an already-resolved Table object; no extra catalog round-trip is needed.
+ case CreateTableLike(
+ ResolvedIdentifier(catalog, ident),
+ ResolvedTable(_, _, table, _),
+ fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
+ CreateTableLikeExec(
+ catalog.asTableCatalog, ident, table, fileFormat, provider, properties, ifNotExists) :: Nil
Review Comment:
Good suggestion. I can refactor to the single pattern you proposed, with the source table resolved in an inner match. This is cleaner and removes the duplication.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4092135671
Agree on that. Okay since we got consensus, I will proceed with this direction and update this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2978443477
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
Review Comment:
I removed the resolved provider inherited from the source table and to be strict with user-overrides info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2976708547
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
Review Comment:
"USING clause" is listed inside the parenthetical as a user override, then "the resolved provider" appears after the closing paren as a seemingly separate item, but they refer to the same concept. Also, the resolved provider can be inherited from the source table when no USING clause is given (see `CreateTableLikeExec` lines 63-73), so "user-specified overrides only" is not strictly accurate.
```suggestion
* The {@code tableInfo} parameter contains user-specified overrides
* (TBLPROPERTIES, LOCATION), the resolved provider (from the USING clause or inherited
* from the source table), and the current user as owner.
```
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/CreateTableLikeSuite.scala:
##########
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector
+
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, InMemoryCatalog}
+import org.apache.spark.sql.types.{CharType, IntegerType, LongType, StringType, VarcharType}
+
+class CreateTableLikeSuite extends DatasourceV2SQLBase {
+
+ private def testCatalog: InMemoryCatalog =
+ catalog("testcat").asInstanceOf[InMemoryCatalog]
+
+ private def testCatalog2: InMemoryCatalog =
+ catalog("testcat2").asInstanceOf[InMemoryCatalog]
+
+ // -------------------------------------------------------------------------
+ // Basic V2 path
+ // -------------------------------------------------------------------------
+
+ test("v2 target, v1 source: schema and partitioning are copied") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet PARTITIONED BY (data)")
+ sql("CREATE TABLE testcat.dst LIKE src")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ assert(dst.schema() === CatalogV2Util.v2ColumnsToStructType(dst.columns()))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ // partition column encoded in partitioning (identity transform on the partition column)
+ assert(dst.partitioning.nonEmpty)
+ }
+ }
+
+ test("v2 target, v2 source: pure v2 path") {
+ withTable("testcat.src", "testcat.dst") {
+ sql("CREATE TABLE testcat.src (id bigint, data string) USING foo")
+ sql("CREATE TABLE testcat.dst LIKE testcat.src")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+
+ test("cross-catalog: source in testcat, target in testcat2") {
+ withTable("testcat.src", "testcat2.dst") {
+ sql("CREATE TABLE testcat.src (id bigint, name string) USING foo")
+ sql("CREATE TABLE testcat2.dst LIKE testcat.src")
+
+ val dst = testCatalog2.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "name"))
+ }
+ }
+
+ test("3-part name: catalog.namespace.table for both target and source") {
+ withTable("testcat.ns.src", "testcat2.ns.dst") {
+ sql("CREATE NAMESPACE IF NOT EXISTS testcat.ns")
+ sql("CREATE NAMESPACE IF NOT EXISTS testcat2.ns")
+ sql("CREATE TABLE testcat.ns.src (id bigint, data string) USING foo")
+ sql("CREATE TABLE testcat2.ns.dst LIKE testcat.ns.src")
+
+ val dst = testCatalog2.loadTable(Identifier.of(Array("ns"), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // IF NOT EXISTS
+ // -------------------------------------------------------------------------
+
+ test("IF NOT EXISTS: second call is silent when table already exists") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint) USING parquet")
+ sql("CREATE TABLE testcat.dst LIKE src")
+ // Should not throw
+ sql("CREATE TABLE IF NOT EXISTS testcat.dst LIKE src")
+ }
+ }
+
+ test("without IF NOT EXISTS, duplicate create throws TableAlreadyExistsException") {
+ withTable("src", "testcat.dst") {
+ sql("CREATE TABLE src (id bigint) USING parquet")
+ sql("CREATE TABLE testcat.dst LIKE src")
+ intercept[TableAlreadyExistsException] {
+ sql("CREATE TABLE testcat.dst LIKE src")
+ }
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Views as source
+ // -------------------------------------------------------------------------
+
+ test("persistent view as source, v2 target") {
+ withTable("src", "testcat.dst") {
+ withView("v") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet")
+ sql("INSERT INTO src VALUES (1, 'a')")
+ sql("CREATE VIEW v AS SELECT * FROM src")
+ sql("CREATE TABLE testcat.dst LIKE v")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+ }
+
+ test("temp view as source, v2 target") {
+ withTable("src", "testcat.dst") {
+ withTempView("tv") {
+ sql("CREATE TABLE src (id bigint, data string) USING parquet")
+ sql("CREATE TEMP VIEW tv AS SELECT id, data FROM src")
+ sql("CREATE TABLE testcat.dst LIKE tv")
+
+ val dst = testCatalog.loadTable(Identifier.of(Array(), "dst"))
+ val columnNames = dst.columns().map(_.name)
+ assert(columnNames === Array("id", "data"))
+ }
+ }
+ }
+
+ // -------------------------------------------------------------------------
+ // Property / provider behavior
+ // -------------------------------------------------------------------------
+
+ test("source TBLPROPERTIES are copied to target when connector implements createTableLike") {
+ // InMemoryTableCatalog overrides createTableLike to merge source properties into the target,
+ // demonstrating connector-specific copy semantics. Connectors that do not override
+ // createTableLike fall back to createTable and do not copy source properties.
Review Comment:
The default `createTableLike` in `TableCatalog` throws `UnsupportedOperationException` — it does not fall back to `createTable`.
```suggestion
// demonstrating connector-specific copy semantics. Connectors that do not override
// createTableLike will throw UnsupportedOperationException.
```
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -240,6 +240,20 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
qualifyLocInTableSpec(tableSpec), options, ifNotExists) :: Nil
}
+ // CREATE TABLE ... LIKE ... for a v2 catalog target.
+ // Source is an already-resolved Table object; no extra catalog round-trip is needed.
+ // Views are wrapped in V1Table so the exec can extract schema and provider uniformly.
+ case CreateTableLike(
+ ResolvedIdentifier(catalog, ident), source,
+ fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
+ val table = source match {
+ case ResolvedTable(_, _, t, _) => t
+ case ResolvedPersistentView(_, _, meta) => V1Table(meta)
+ case ResolvedTempView(_, meta) => V1Table(meta)
+ }
+ CreateTableLikeExec(catalog.asTableCatalog, ident, table,
+ fileFormat.locationUri, provider, properties, ifNotExists) :: Nil
Review Comment:
Every other CREATE/REPLACE operation in this file qualifies the LOCATION via `qualifyLocInTableSpec` (lines 229, 237, 240, 275, 278, 290, 300), but `CreateTableLike` passes the raw `fileFormat.locationUri` without qualification. If a user specifies a relative LOCATION, it won't be resolved to an absolute path.
Consider qualifying it here, e.g.:
```scala
val qualifiedLocation = fileFormat.locationUri.map { uri =>
if (uri.isAbsolute) uri
else CatalogUtils.makeQualifiedTablePath(new Path(uri), session.sessionState.conf).toUri
}
CreateTableLikeExec(catalog.asTableCatalog, ident, table,
qualifiedLocation, provider, properties, ifNotExists) :: Nil
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2942063354
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
Review Comment:
Correct on both. sourceTable: Table is the V2 Table interface, which can be any implementation. For session catalog sources, ResolveRelations wraps the CatalogTable in a V1Table, which implements Table. So V1→V2 works: the source is a V1Table and we handle it explicitly in the match block at line 57 to preserve CHAR/VARCHAR types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2942077606
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
Review Comment:
Yes — "v2 target, v1 source: schema and partitioning are copied" tests V1 source (default.src in session catalog) → V2 target (testcat.dst). The "cross-catalog" and "3-part name" tests cover V2→V2 across catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944062410
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala:
##########
@@ -240,6 +240,33 @@ class DataSourceV2Strategy(session: SparkSession) extends Strategy with Predicat
qualifyLocInTableSpec(tableSpec), options, ifNotExists) :: Nil
}
+ // CREATE TABLE ... LIKE ... for a v2 catalog target.
+ // Source is an already-resolved Table object; no extra catalog round-trip is needed.
+ case CreateTableLike(
+ ResolvedIdentifier(catalog, ident),
+ ResolvedTable(_, _, table, _),
+ fileFormat: CatalogStorageFormat, provider, properties, ifNotExists) =>
+ CreateTableLikeExec(
+ catalog.asTableCatalog, ident, table, fileFormat, provider, properties, ifNotExists) :: Nil
Review Comment:
Good suggestion. I can refactor to the single pattern you proposed, with the source table resolved in an inner match. This is cleaner and removes the duplication. I wonder if we have precedents of DDL which consolidate both V1 and V2 commands?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4087691989
How about adding a capability-gated `createTableLike` to `TableCatalog`:
```java
// New capability
enum TableCatalogCapability {
SUPPORT_CREATE_TABLE_LIKE
}
// New default method in TableCatalog
default Table createTableLike(
Identifier ident,
Identifier sourceIdent,
TableInfo overrides) {
throw new UnsupportedOperationException(
name() + " does not support CREATE TABLE LIKE");
}
```
And in Spark's execution layer:
```java
if (catalog.capabilities().contains(SUPPORT_CREATE_TABLE_LIKE)) {
// Connector handles everything: loads source, filters internal props, applies overrides
catalog.createTableLike(ident, sourceIdent, overrides)
} else {
// Fallback: Spark loads source, builds merged TableInfo, calls createTable
val source = catalog.loadTable(sourceIdent)
val merged = buildTableInfoFromSource(source, overrides)
catalog.createTable(ident, merged)
}
```
This gives connectors like Delta full control over internal property filtering/ protocol inheritance in while still providing a reasonable fallback for simpler connectors that don't opt in. The fallback path handles the "good enough" case; connectors with complex internal state (Delta, Iceberg) implement the capability for correctness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964852983
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
Same question for properties too. If we copied partitioning, why didn't we copy properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2976694336
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
+ * It does NOT contain schema, partitioning, properties, or constraints from the source table.
+ * Connectors must read all source metadata directly from {@code sourceTable}, including
+ * columns ({@link Table#columns()}), partitioning ({@link Table#partitioning()}),
+ * constraints ({@link Table#constraints()}), and format-specific properties
+ * ({@link Table#properties()}).
+ * <p>
+ * The default implementation throws {@link UnsupportedOperationException}. Connectors that
+ * support {@code CREATE TABLE ... LIKE ...} must override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides only: TBLPROPERTIES, LOCATION, resolved provider,
+ * and owner; source schema, partitioning, and constraints are NOT included
+ * @param sourceTable the resolved source table; connectors read schema, partitioning,
+ * constraints, properties, and any format-specific metadata from this object
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @throws UnsupportedOperationException If the catalog does not support CREATE TABLE LIKE
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
Review Comment:
1. shall we move `sourceTable` before `tableInfo`?
2. shall we rename tableInfo as `userSpecifiedOverrides` or `userSpecifiedTableInfo`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2978169941
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,38 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
Review Comment:
Actually I'm thinking if we should keep strict to user specified overrides only for the `userSpecifiedOverrides` parameter? i.e., only provide the resolved provider from the USING clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944057635
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -293,6 +293,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
c
}
+ // For CREATE TABLE LIKE, use the v1 command if both the target and source are in the session
+ // catalog (or a V1-compatible catalog extension). If source is in a different catalog, fall
+ // through to the V2 execution path (CreateTableLikeExec via DataSourceV2Strategy).
+ case CreateTableLike(
Review Comment:
Good catch. When a connector like Iceberg overrides the session catalog, the target resolves through ResolvedV1Identifier (since supportsV1Command returns true for the session catalog), but the source — a native Iceberg Table — does NOT match ResolvedV1TableOrViewIdentifier (which requires V1Table). So ResolveSessionCatalog falls through and CreateTableLikeExec handles it, passing the Iceberg Table directly. The target createTable call goes to V2SessionCatalog, which delegates to the Iceberg catalog extension. This should work, but deserves a test. I can add one if you'd like.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4072510746
> BTW, consider unifying to a single CreateTableLikeExec — The current PR keeps two execution paths: V1 fallback via CreateTableLikeCommand (for V1-V1 cases) and the new CreateTableLikeExec (for V2 targets). The test "v2 source, v1 target" already proves CreateTableLikeExec works for session catalog targets via V2SessionCatalog.
I considered this, but prefer to keep the separation consistent with how every other V2 DDL command in Spark is structured — ResolveSessionCatalog routes session catalog targets back to V1 commands, and DataSourceV2Strategy handles the rest. Unifying would require CreateTableLikeExec to absorb Hive serde semantics (STORED AS, ROW FORMAT, etc.) that properly belong to CreateTableLikeCommand. The V2→V1 test passing with a simple parquet table doesn't cover those cases. I think we'd rather keep the boundary clean and follow the established pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941466438
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
Review Comment:
Do we have V1 -> V2 within as well across catalog tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941570258
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -293,6 +293,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
c
}
+ // For CREATE TABLE LIKE, use the v1 command if both the target and source are in the session
+ // catalog (or a V1-compatible catalog extension). If source is in a different catalog, fall
+ // through to the V2 execution path (CreateTableLikeExec via DataSourceV2Strategy).
+ case CreateTableLike(
Review Comment:
An example is Iceberg [session catalog](https://github.com/apache/iceberg/blob/8b7f6d773f99d4b581f151efbd871c8965f2b3cb/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSessionCatalog.java#L64).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964852983
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
Same question for properties too. If we copied partitioning, why didn't we copy properties? This isn't strictly overrides or am I missing something?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2967084578
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,36 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause) — it does NOT contain properties copied from the
+ * source table. Connectors that want to copy source-format-specific metadata (e.g. Delta
+ * protocol version, Iceberg sort order, format version) should read it directly from
+ * {@code sourceTable}.
+ * <p>
+ * The default implementation falls back to {@link #createTable(Identifier, TableInfo)}, which
+ * creates a new table using only the schema and partitioning extracted from {@code sourceTable}
+ * by Spark, plus the user-specified overrides in {@code tableInfo}. Connectors with
+ * format-specific copy semantics should override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides (TBLPROPERTIES, LOCATION), resolved provider, and
+ * current user as owner; source TBLPROPERTIES are NOT bulk-copied
+ * @param sourceTable the resolved source table whose metadata is being copied
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ return createTable(ident, tableInfo);
Review Comment:
I'm going to remove the default implementation now.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
Removed them as no default implementation now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2968060175
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,39 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause), the resolved provider, and the current user as owner.
+ * It does NOT contain schema, partitioning, properties, or constraints from the source table.
+ * Connectors must read all source metadata directly from {@code sourceTable}, including
+ * columns ({@link Table#columns()}), partitioning ({@link Table#partitioning()}),
+ * constraints ({@link Table#constraints()}), and format-specific properties
+ * ({@link Table#properties()}).
+ * <p>
+ * There is no default implementation. Connectors that support {@code CREATE TABLE ... LIKE ...}
+ * must override this method. Connectors that do not override it will throw
+ * {@link UnsupportedOperationException} when the command is issued against their catalog.
Review Comment:
The method IS declared with the `default` keyword and has a body, so "There is no default implementation" reads as contradictory. Suggested rewording:
```suggestion
* The default implementation throws {@link UnsupportedOperationException}. Connectors that
* support {@code CREATE TABLE ... LIKE ...} must override this method.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941460522
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider and location are added if determined above.
+ val locationProp: Option[(String, String)] =
+ fileFormat.locationUri.map(uri =>
+ TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp
+
+ try {
+ // Constraints from the source table are intentionally NOT copied for several reasons:
+ // 1. V1 tables (CatalogTable) have no constraint objects — CHECK, PRIMARY KEY,
+ // UNIQUE and FOREIGN KEY are V2-only concepts (introduced in Spark 4.1.0).
+ // CreateTableLikeCommand had nothing to copy, so not copying here preserves
+ // behavioral parity with the V1 path.
+ // 2. ForeignKey constraints carry a refTable Identifier bound to the source catalog;
+ // copying them to a different catalog creates dangling cross-catalog references.
+ // 3. Constraint names must be unique within a namespace; blindly copying them risks
+ // collisions with existing constraints in the target namespace.
+ // 4. Validation status (VALID/INVALID/UNVALIDATED) is tied to the source table's
+ // existing data and is meaningless on a newly created empty target table.
+ // 5. NOT NULL is already captured in Column.nullable() and copied via withColumns.
+ // 6. Consistent with PostgreSQL semantics: CREATE TABLE LIKE does not include
+ // constraints by default; users must explicitly opt in via INCLUDING CONSTRAINTS.
+ // If constraint copying is desired, use ALTER TABLE ADD CONSTRAINT after creation.
+ // If we wanted to support them in the future, the right approach would be to add an
+ // INCLUDING CONSTRAINTS clause (as PostgreSQL does) rather than copying blindly.
+ val tableInfo = new TableInfo.Builder()
Review Comment:
Owner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2943469874
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -293,6 +293,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
c
}
+ // For CREATE TABLE LIKE, use the v1 command if both the target and source are in the session
+ // catalog (or a V1-compatible catalog extension). If source is in a different catalog, fall
+ // through to the V2 execution path (CreateTableLikeExec via DataSourceV2Strategy).
+ case CreateTableLike(
Review Comment:
yea, agree, we should add a test for sessionCatalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941389972
##########
sql/core/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveSessionCatalog.scala:
##########
@@ -293,6 +293,16 @@ class ResolveSessionCatalog(val catalogManager: CatalogManager)
c
}
+ // For CREATE TABLE LIKE, use the v1 command if both the target and source are in the session
+ // catalog (or a V1-compatible catalog extension). If source is in a different catalog, fall
+ // through to the V2 execution path (CreateTableLikeExec via DataSourceV2Strategy).
+ case CreateTableLike(
Review Comment:
What does this mean for DSv2 connectors that override the session catalog?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4068867499
@gengliangwang @cloud-fan, can you folks help review as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2941495574
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Copies schema (columns) and partitioning from `sourceTable`. The following properties of the
+ * source table are intentionally NOT copied (matching v1 behavior):
+ * - Table-level comments
+ * - Source table's TBLPROPERTIES (user-specified `properties` are used instead)
+ * - Statistics, owner, create time
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ fileFormat: CatalogStorageFormat,
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ val resolvedProvider = provider.orElse {
Review Comment:
What does DSv1 do and is it applicable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964782450
##########
sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCatalog.java:
##########
@@ -295,6 +295,36 @@ default Table createTable(Identifier ident, TableInfo tableInfo)
return createTable(ident, tableInfo.columns(), tableInfo.partitions(), tableInfo.properties());
}
+ /**
+ * Create a table in the catalog by copying metadata from an existing source table.
+ * <p>
+ * This method is called for {@code CREATE TABLE ... LIKE ...} statements targeting this catalog.
+ * The {@code tableInfo} parameter contains only the explicit overrides provided by the user
+ * (TBLPROPERTIES, LOCATION, USING clause) — it does NOT contain properties copied from the
+ * source table. Connectors that want to copy source-format-specific metadata (e.g. Delta
+ * protocol version, Iceberg sort order, format version) should read it directly from
+ * {@code sourceTable}.
+ * <p>
+ * The default implementation falls back to {@link #createTable(Identifier, TableInfo)}, which
+ * creates a new table using only the schema and partitioning extracted from {@code sourceTable}
+ * by Spark, plus the user-specified overrides in {@code tableInfo}. Connectors with
+ * format-specific copy semantics should override this method.
+ *
+ * @param ident a table identifier for the new table
+ * @param tableInfo user-specified overrides (TBLPROPERTIES, LOCATION), resolved provider, and
+ * current user as owner; source TBLPROPERTIES are NOT bulk-copied
+ * @param sourceTable the resolved source table whose metadata is being copied
+ * @return metadata for the new table
+ *
+ * @throws TableAlreadyExistsException If a table or view already exists for the identifier
+ * @throws NoSuchNamespaceException If the identifier namespace does not exist (optional)
+ * @since 4.2.0
+ */
+ default Table createTableLike(Identifier ident, TableInfo tableInfo, Table sourceTable)
+ throws TableAlreadyExistsException, NoSuchNamespaceException {
+ return createTable(ident, tableInfo);
Review Comment:
Is this actually safe or should we throw an exception by default?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
This seems inconsistent. It does NOT seem like TableInfo contains only overrides, it seems like it actually copies some state from the source table. And if it copies partitioning, why doesn't it copy constraints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2964799256
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/CreateTableLikeExec.scala:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import java.net.URI
+
+import scala.jdk.CollectionConverters._
+
+import org.apache.spark.internal.LogKeys.TABLE_NAME
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
+import org.apache.spark.sql.catalyst.catalog.{CatalogTableType, CatalogUtils}
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.connector.catalog.{CatalogV2Util, Identifier, Table, TableCatalog, TableInfo, V1Table}
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.errors.QueryCompilationErrors
+
+/**
+ * Physical plan node for CREATE TABLE ... LIKE ... targeting a v2 catalog.
+ *
+ * Calls [[TableCatalog.createTableLike]] so that connectors can implement format-specific copy
+ * semantics (e.g. Delta protocol inheritance, Iceberg sort order and format version). Connectors
+ * that do not override [[TableCatalog.createTableLike]] fall back to [[TableCatalog.createTable]]
+ * with only the schema and partitioning extracted from `sourceTable` by Spark.
+ *
+ * The [[TableInfo]] passed to [[TableCatalog.createTableLike]] contains user-specified overrides
+ * (TBLPROPERTIES, LOCATION), the resolved provider, and the current user as owner. Source
+ * TBLPROPERTIES and constraints are NOT bulk-copied; connectors can read them directly from
+ * [[sourceTable]].
+ */
+case class CreateTableLikeExec(
+ targetCatalog: TableCatalog,
+ targetIdent: Identifier,
+ sourceTable: Table,
+ location: Option[URI],
+ provider: Option[String],
+ properties: Map[String, String],
+ ifNotExists: Boolean) extends LeafV2CommandExec {
+
+ override def output: Seq[Attribute] = Seq.empty
+
+ override protected def run(): Seq[InternalRow] = {
+ if (!targetCatalog.tableExists(targetIdent)) {
+ // 1. Extract columns from source. For V1Table sources use the raw schema so that
+ // CHAR/VARCHAR types are preserved as declared (without internal metadata expansion).
+ val columns = sourceTable match {
+ case v1: V1Table =>
+ val rawSchema = CharVarcharUtils.getRawSchema(v1.catalogTable.schema)
+ CatalogV2Util.structTypeToV2Columns(rawSchema)
+ case _ =>
+ sourceTable.columns()
+ }
+
+ // 2. Extract partitioning from source (includes both partition columns and bucket spec
+ // for V1Table, as V1Table.partitioning encodes both).
+ val partitioning = sourceTable.partitioning
+
+ // 3. Resolve provider: USING clause overrides, else copy from source.
+ // The source provider is inherited so that the target table uses the same format,
+ // matching V1 CreateTableLikeCommand behavior. Whether the target catalog validates
+ // or uses this property is catalog-specific (e.g. V2SessionCatalog validates it).
+ val resolvedProvider = provider.orElse {
+ sourceTable match {
+ case v1: V1Table if v1.catalogTable.tableType == CatalogTableType.VIEW =>
+ // When the source is a view, default to the session's default data source.
+ // This matches V1 CreateTableLikeCommand behavior.
+ Some(session.sessionState.conf.defaultDataSourceName)
+ case v1: V1Table =>
+ v1.catalogTable.provider
+ case _ =>
+ Option(sourceTable.properties.get(TableCatalog.PROP_PROVIDER))
+ }
+ }
+
+ // 4. Build final properties. User-specified TBLPROPERTIES are used as-is; source table
+ // properties are NOT copied. Provider, location, and owner are added if applicable.
+ val locationProp: Option[(String, String)] =
+ location.map(uri => TableCatalog.PROP_LOCATION -> CatalogUtils.URIToString(uri))
+
+ val finalProps = CatalogV2Util.withDefaultOwnership(
+ properties ++
+ resolvedProvider.map(TableCatalog.PROP_PROVIDER -> _) ++
+ locationProp)
+
+ try {
+ // The TableInfo passed to createTableLike contains user-specified overrides
+ // (TBLPROPERTIES, LOCATION), the resolved provider (from USING clause if specified,
+ // otherwise inherited from the source), and the current user as owner. Source
+ // TBLPROPERTIES are NOT bulk-copied; connectors that override createTableLike can
+ // read source metadata, including properties and constraints, directly from sourceTable.
+ val tableInfo = new TableInfo.Builder()
+ .withColumns(columns)
+ .withPartitions(partitioning)
Review Comment:
This seems inconsistent. It does NOT seem like TableInfo contains only overrides, it seems like it actually copies some but not all state from the source table, making it really confusing. And if it copies partitioning, why doesn't it copy constraints?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2976717857
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala:
##########
@@ -615,6 +615,35 @@ case class CreateTable(
}
}
+/**
+ * Create a new table with the same schema/partitioning as an existing table or view,
+ * for use with a v2 catalog.
+ *
+ * @param name Target table identifier. Starts as UnresolvedIdentifier, resolved to
+ * ResolvedIdentifier by ResolveCatalogs.
+ * @param source Source table or view. Starts as UnresolvedTableOrView, resolved to
+ * ResolvedTable / ResolvedPersistentView / ResolvedTempView by ResolveRelations.
+ * @param fileFormat User-specified STORED AS / ROW FORMAT (Hive-style). Empty if not specified.
+ * @param provider User-specified USING provider. None if not specified.
+ * @param properties User-specified TBLPROPERTIES.
+ * @param ifNotExists IF NOT EXISTS flag.
+ */
+case class CreateTableLike(
+ name: LogicalPlan,
+ source: LogicalPlan,
+ fileFormat: CatalogStorageFormat,
Review Comment:
`CatalogStorageFormat` is a V1 catalog concept (Hive serde, input/output format, etc.) and no other V2 command in this file uses it. The V2 execution path (`DataSourceV2Strategy` → `CreateTableLikeExec`) only extracts `locationUri` from it — the serde fields are silently discarded. Only the V1 fallback in `ResolveSessionCatalog` uses the full `CatalogStorageFormat`.
Consider replacing this with `location: Option[String]` in the V2 plan and reconstructing the full `CatalogStorageFormat` only in the `ResolveSessionCatalog` V1 fallback path. This would remove the V1 catalog dependency from the V2 logical plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #54809:
URL: https://github.com/apache/spark/pull/54809#discussion_r2944065106
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLParserSuite.scala:
##########
@@ -721,83 +721,76 @@ class DDLParserSuite extends AnalysisTest with SharedSparkSession {
}
test("create table like") {
- val v1 = "CREATE TABLE table1 LIKE table2"
+ // Helper to extract fields from the new CreateTableLike unresolved plan.
+ // The parser now emits CreateTableLike (v2 logical plan) instead of
+ // CreateTableLikeCommand, so both name and source are unresolved identifiers.
Review Comment:
Good catch, I'll apply the suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "sarutak (via GitHub)" <gi...@apache.org>.
sarutak commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4088687974
I think @aokolnychyi's `createTableLike(ident, tableInfo, sourceTable)` with overrides-only `tableInfo` works well too. Passing the resolved `Table` object lets connectors use `instanceof` for format-specific metadata (Iceberg sort order, Delta protocol version, etc.), and the overrides-only `tableInfo` keeps the contract clean.
If the need for a source-side contract like `tableInfoForLike()` becomes clear in practice, it can be added later as a default method on `Table` without breaking existing implementations. No reason to block on it now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-33902][SQL] Support CREATE TABLE LIKE for V2 [spark]
Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #54809:
URL: https://github.com/apache/spark/pull/54809#issuecomment-4091579800
@sarutak I see. Using `sourceTable` instead of `sourceIdent` makes sense.
@aokolnychyi sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org