You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2025/09/06 14:36:24 UTC
[PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Zouxxyy opened a new pull request, #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646
<!--
Thank you for submitting a pull request! Here are some tips:
1. For first-time contributors, please read our contributing guide:
https://github.com/apache/incubator-gluten/blob/main/CONTRIBUTING.md
2. If necessary, create a GitHub issue for discussion beforehand to avoid duplicate work.
3. If the PR is specific to a single backend, include [VL] or [CH] in the PR title to indicate the
Velox or ClickHouse backend, respectively.
4. If the PR is not ready for review, please mark it as a draft.
-->
## What changes are proposed in this pull request?
<!--
Provide a clear and concise description of the changes introduced in this PR.
Ensure the PR description aligns with the code changes, especially after updates.
If applicable, include "Fixes #<GitHub_Issue_ID>" to automatically close the corresponding issue
when the PR is merged.
-->
## How was this patch tested?
<!--
Describe how the changes were tested, if applicable.
Include new tests to validate the functionality, if necessary.
For UI-related changes, attach screenshots to demonstrate the updates.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262302477
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262400914
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262315995
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on code in PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#discussion_r2333188371
##########
backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/enhanced/VeloxIcebergSuite.scala:
##########
@@ -190,4 +192,29 @@ class VeloxIcebergSuite extends IcebergSuite {
assert(result(0).get(0) == 2)
}
}
+
+ test("iceberg create table as select") {
+ withTable("iceberg_tb1", "iceberg_tb2") {
+ spark.sql("""
+ |create table iceberg_tb1 (a int, pt int) using iceberg
+ |partitioned by (pt)
+ |""".stripMargin)
+
+ spark.sql("insert into table iceberg_tb1 values (1, 1), (2, 2)")
+
+ // CTAS
+ val sqlStr = """
+ |create table iceberg_tb2 using iceberg
+ |partitioned by (pt)
+ |as select * from iceberg_tb1
+ |""".stripMargin
+
+ TestUtils.checkExecutedPlanContains[VeloxIcebergAppendDataExec](spark, sqlStr)
Review Comment:
Could we use assert(
getExecutedPlan(df).exists(
plan => plan.find(_.isInstanceOf[VeloxIcebergAppendDataExec]).isDefined))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3270319141
VeloxIcebergSuite runs on Spark34 and Spark35, so this feature is supported in both Spark 34 and 35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262451713
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on code in PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#discussion_r2333458607
##########
shims/spark34/src/main/scala/org/apache/spark/sql/extension/RewriteCreateTableAsSelect.scala:
##########
@@ -0,0 +1,202 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.extension
+
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogUtils
+import org.apache.spark.sql.catalyst.expressions.{Attribute, FileSourceMetadataAttribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.util.{CharVarcharUtils, METADATA_COL_ATTR_KEY}
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.CatalogHelper
+import org.apache.spark.sql.connector.catalog.CatalogV2Util.isSessionCatalog
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.execution.{SparkPlan, SparkStrategy}
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, LeafV2CommandExec}
+import org.apache.spark.sql.internal.StaticSQLConf.WAREHOUSE_PATH
+import org.apache.spark.sql.types.{MetadataBuilder, StructType}
+import org.apache.spark.util.Utils
+
+import scala.collection.JavaConverters._
+
+case class RewriteCreateTableAsSelect(session: SparkSession) extends SparkStrategy {
+
+ override def apply(plan: LogicalPlan): Seq[SparkPlan] = {
+ plan match {
+ case CreateTableAsSelect(
+ ResolvedIdentifier(catalog, ident),
+ parts,
+ query,
+ tableSpec: TableSpec,
+ options,
+ ifNotExists,
+ analyzedQuery) if analyzedQuery.isDefined && !supportsV1Command(catalog) =>
+ catalog match {
+ case staging: StagingTableCatalog =>
+ AtomicCreateTableAsSelectExec(
+ staging,
+ ident,
+ parts,
+ analyzedQuery.get,
+ qualifyLocInTableSpec(tableSpec),
+ options,
+ ifNotExists) :: Nil
+ case _ =>
+ CreateTableAsSelectExec(
+ catalog.asTableCatalog,
+ ident,
+ parts,
+ analyzedQuery.get,
+ qualifyLocInTableSpec(tableSpec),
+ options,
+ ifNotExists) :: Nil
+ }
+
+ case _ => Nil
+ }
+ }
+
+ private def supportsV1Command(catalog: CatalogPlugin): Boolean = {
+ isSessionCatalog(catalog) && catalog.isInstanceOf[CatalogExtension]
+ }
+
+ private def qualifyLocInTableSpec(tableSpec: TableSpec): TableSpec = {
+ tableSpec.copy(location = tableSpec.location.map(makeQualifiedDBObjectPath))
+ }
+
+ private def makeQualifiedDBObjectPath(location: String): String = {
+ CatalogUtils.makeQualifiedDBObjectPath(
+ session.sharedState.conf.get(WAREHOUSE_PATH),
+ location,
+ session.sharedState.hadoopConf)
+ }
+}
+
+/** Port from Spark 3.5. */
+case class AtomicCreateTableAsSelectExec(
Review Comment:
I see this logic now. so this rewrite is only required for spark-34, spark-35 already provide this support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262388998
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#discussion_r2333306937
##########
backends-velox/src-iceberg/test/scala/org/apache/gluten/execution/enhanced/VeloxIcebergSuite.scala:
##########
@@ -190,4 +192,29 @@ class VeloxIcebergSuite extends IcebergSuite {
assert(result(0).get(0) == 2)
}
}
+
+ test("iceberg create table as select") {
+ withTable("iceberg_tb1", "iceberg_tb2") {
+ spark.sql("""
+ |create table iceberg_tb1 (a int, pt int) using iceberg
+ |partitioned by (pt)
+ |""".stripMargin)
+
+ spark.sql("insert into table iceberg_tb1 values (1, 1), (2, 2)")
+
+ // CTAS
+ val sqlStr = """
+ |create table iceberg_tb2 using iceberg
+ |partitioned by (pt)
+ |as select * from iceberg_tb1
+ |""".stripMargin
+
+ TestUtils.checkExecutedPlanContains[VeloxIcebergAppendDataExec](spark, sqlStr)
Review Comment:
I tried it, seemed didn't work. Maybe it's because `AtomicCreateTableAsSelect` constructs another AppendData exec runtime. The `df.queryExecution.executedPlan` just like this
```text
CommandResult <empty>
+- AtomicCreateTableAsSelect org.apache.iceberg.spark.SparkCatalog@69c263f8, default.iceberg_tb2, [identity(pt)], Project [a#10, pt#11], TableSpec(Map(),Some(iceberg),Map(),None,None,None,false), false
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "jinchengchenghh (via GitHub)" <gi...@apache.org>.
jinchengchenghh merged PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262328731
@jinchengchenghh I tried to offload the CTAS command, and I found two main reasons that prevented me from doing so:
1. Many classes need to be copied, and it is difficult to validate fallback logic:
This is because we can only construct the v2 `write` after creating the table, which is required for performing the validation. By that point, it's already too late to perform proper fallback.
2. Spark 3.5 alreadly supports columnar v2 write for CTAS after this PR was introduced https://github.com/apache/spark/pull/40734, this PR is essentially a copy of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3269562574
> @Zouxxyy @jinchengchenghh could we also enable this in Spark-35 code path?
If we run VeloxIcebergSuite on Spark35, it should work too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262350459
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3290325614
@jinchengchenghh @zhouyuan Could you have a look again? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "zhouyuan (via GitHub)" <gi...@apache.org>.
zhouyuan commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3265664096
@Zouxxyy @jinchengchenghh could we also enable this in Spark-35 code path?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3264355031
Run Gluten Clickhouse CI on x86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [PR] [GLUTEN-10625][CORE] Add RewriteCreateTableAsSelect for spark34 [incubator-gluten]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #10646:
URL: https://github.com/apache/incubator-gluten/pull/10646#issuecomment-3262301799
https://github.com/apache/incubator-gluten/issues/10625
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org