You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2021/12/24 13:07:04 UTC
[GitHub] [iceberg] aokolnychyi opened a new pull request #3804: Spark: Implement copy-on-write MERGE
aokolnychyi opened a new pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804
This PR implements copy-on-write MERGE in Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775012012
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
##########
@@ -127,13 +130,19 @@ class IcebergSparkSqlExtensionsParser(delegate: ParserInterface) extends ParserI
}
}
- private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan match {
+ private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsDown {
case DeleteFromTable(UnresolvedIcebergTable(aliasedTable), condition) =>
DeleteFromIcebergTable(aliasedTable, condition)
+
case UpdateTable(UnresolvedIcebergTable(aliasedTable), assignments, condition) =>
UpdateIcebergTable(aliasedTable, assignments, condition)
- case _ =>
- plan
+
+ case MergeIntoTable(UnresolvedIcebergTable(aliasedTable), source, cond, matchedActions, notMatchedActions) =>
+ // cannot construct MergeIntoIcebergTable right away as MERGE operations require special resolution
Review comment:
This comment is important, @rdblue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776482946
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
+
+ private lazy val analyzer: Analyzer = spark.sessionState.analyzer
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ case m @ UnresolvedMergeIntoIcebergTable(targetTable, sourceTable, context)
+ if targetTable.resolved && sourceTable.resolved && m.duplicateResolved =>
+
+ val resolvedMatchedActions = context.matchedActions.map {
+ case DeleteAction(cond) =>
+ val resolvedCond = cond.map(resolveCond("DELETE", _, m))
+ DeleteAction(resolvedCond)
+
+ case UpdateAction(cond, assignments) =>
+ val resolvedCond = cond.map(resolveCond("UPDATE", _, m))
+ // the update action can access columns from both target and source tables
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = false)
+ UpdateAction(resolvedCond, resolvedAssignments)
+
+ case UpdateStarAction(updateCondition) =>
+ val resolvedUpdateCondition = updateCondition.map(resolveCond("UPDATE", _, m))
+ val assignments = targetTable.output.map { attr =>
+ Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
+ }
+ // for UPDATE *, the value must be from the source table
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
+ UpdateAction(resolvedUpdateCondition, resolvedAssignments)
+
+ case o => o
Review comment:
This is an unsupported action, right? Why just pass it through?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776887805
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
Review comment:
Looks fairly accurate to the original.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776892305
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
Oh, I see. That's why there's an extra `true` here and why it projects `readAttrs` below. Maybe add a comment to explain or fix up the outputs/conditions in a method like before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776893212
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
##########
@@ -127,13 +130,19 @@ class IcebergSparkSqlExtensionsParser(delegate: ParserInterface) extends ParserI
}
}
- private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan match {
+ private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsDown {
case DeleteFromTable(UnresolvedIcebergTable(aliasedTable), condition) =>
DeleteFromIcebergTable(aliasedTable, condition)
+
case UpdateTable(UnresolvedIcebergTable(aliasedTable), assignments, condition) =>
UpdateIcebergTable(aliasedTable, assignments, condition)
- case _ =>
- plan
+
+ case MergeIntoTable(UnresolvedIcebergTable(aliasedTable), source, cond, matchedActions, notMatchedActions) =>
+ // cannot construct MergeIntoIcebergTable right away as MERGE operations require special resolution
Review comment:
This looks good to me. I think it works fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r780561731
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
There won't be an `InsertStarAction` or `UpdateStartAction` because `m.resolved` must be true and the plan is not resolved if there are any star actions: https://github.com/apache/iceberg/pull/3804/files#diff-aca4581053978b4b0146677000184c3200de1c0ed13505e58846659ece456538R37-R47
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
+ case d: DeleteAction =>
+ d
+ case _ =>
+ throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
+ }
+
+ val alignedNotMatchedActions = m.notMatchedActions.map {
+ case i @ InsertAction(_, assignments) =>
Review comment:
See above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775012153
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775016317
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ context: MergeIntoContext) extends BinaryCommand {
+
+ def duplicateResolved: Boolean = targetTable.outputSet.intersect(sourceTable.outputSet).isEmpty
Review comment:
Essentially, I checked all places that resolve `UnresolvedRelation` both in the session and non-session catalogs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776891720
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
Why is `rewriteMatchedActions` no longer needed in this version? Is that because this may not be optimized yet and the condition could be `1 = 1` instead of a `true` literal?
The method served a couple purposes before. It would prune any unnecessary match actions after `true` and it would insert a `true` action so that the target row was always emitted. The last seems important here since we need the target row if it is not modified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782361040
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
Yeah, it is a little bit tricky. I do have a comment above but let me see what can be done to simplify this place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782365679
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
Review comment:
Will add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#issuecomment-1013438180
Thanks for reviewing, @rdblue @kbendick!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784126056
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
+ val currentRowId = rowIdProj.apply(inputRow)
+ if (currentRowId == lastMatchedRowId) {
+ throw new SparkException(
+ "The ON search condition of the MERGE statement matched a single row from " +
+ "the target table with multiple rows of the source table. This could result " +
+ "in the target row being operated on more than once with an update or delete " +
+ "operation and is not allowed.")
+ }
+ lastMatchedRowId = currentRowId.copy()
+ } else {
+ lastMatchedRowId = null
+ }
+
+ if (emitNotMatchedTargetRows && !isSourceRowPresent) {
Review comment:
Well, the way it is written now, we would need to evaluate the is target/source predicate again and will have an extra call for each row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784128938
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData.byPosition(r, project)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, Nil))
+
+ // merge rows as there are multiple not matched actions
+ val mergeRows = MergeRows(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = Nil,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false,
+ output = buildMergeRowsOutput(Nil, notMatchedOutputs, r.output),
+ joinPlan)
+
+ AppendData.byPosition(r, mergeRows)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over
+ val matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral
+ val matchedOutputs = matchedActions.map(actionOutput(_, metadataAttrs)) :+ readAttrs
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, metadataAttrs))
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ val mergeRows = MergeRows(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = if (notMatchedActions.isEmpty) TrueLiteral else IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedConditions,
+ matchedOutputs = matchedOutputs,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = readAttrs,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true,
+ output = buildMergeRowsOutput(matchedOutputs, notMatchedOutputs, readAttrs),
+ joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def actionOutput(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Seq[Expression] = {
+
+ clause match {
+ case u: UpdateAction =>
+ u.assignments.map(_.value) ++ metadataAttrs
+
+ case _: DeleteAction =>
+ Nil
Review comment:
@rdblue, I had to replace `Seq[Option[Seq[Expression]]]` with `Seq[Seq[Expression]]` so that optimizer won't ignore the output expressions. That's why I switched to `Nil` for `DeleteAction` instead of using `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Narcasserun edited a comment on pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
Narcasserun edited a comment on pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#issuecomment-1001463617
Great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775010207
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
Review comment:
It took me a while to overcome the resolution issues. In short, MERGE operations require custom resolution and Spark handles `MergeIntoTable` in a special way. We, on the other hand, have a custom MERGE node that Spark treats like a regular node. That's why I had to introduce `UnresolvedMergeIntoIcebergTable` to hide expressions and add a separate rule to do the resolution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776895228
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
And I also saw the later comment that the optimizer doesn't run on the expressions. That's concerning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776894982
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
Review comment:
Can we test a case where the target table small and the source table is large to attempt to trigger the left broadcast?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r834842493
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData.byPosition(r, project)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, Nil))
+
+ // merge rows as there are multiple not matched actions
+ val mergeRows = MergeRows(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = Nil,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false,
+ output = buildMergeRowsOutput(Nil, notMatchedOutputs, r.output),
+ joinPlan)
+
+ AppendData.byPosition(r, mergeRows)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
Review comment:
I don't think it is related. Predicate pushdown should work if you are using 0.13.1 or master and have a predicate in the ON condition of a MERGE operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782361371
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
+
+ val nullabilityMap = attrs.indices.map { index =>
+ index -> outputs.exists(output => output(index).nullable)
+ }.toMap
+
+ val output = attrs.zipWithIndex.map { case (attr, index) =>
Review comment:
Oh, yeah. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782362862
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Copied from `LogicalPlan`. Shall we just use `super.resolved && startActionsResolved`?
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Copied from `LogicalPlan`. Shall we just use `super.resolved && starActionsResolved`?
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
+ }
+
+ private def startActionsResolved: Boolean = {
Review comment:
Yep, typo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782369756
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
Maybe? I'm not sure but I think we should make it so the optimizer runs. Creating a plan with real children sounds like a good way to do that to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776894877
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
Review comment:
It looks good to me and is passing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775013951
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -310,39 +309,6 @@ public void testMergeWithMultipleUpdatesForTargetRow() {
sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
}
- @Test
- public void testMergeWithDisabledCardinalityCheck() {
Review comment:
We introduced this table property as we expected the cardinality check to be too expensive. In particular, we thought we would need a separate job for merge-on-read. However, the current implementation is cheap and does not require any extra jobs so I am not sure we need that property...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784124798
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
+ val currentRowId = rowIdProj.apply(inputRow)
+ if (currentRowId == lastMatchedRowId) {
+ throw new SparkException(
+ "The ON search condition of the MERGE statement matched a single row from " +
+ "the target table with multiple rows of the source table. This could result " +
+ "in the target row being operated on more than once with an update or delete " +
+ "operation and is not allowed.")
+ }
+ lastMatchedRowId = currentRowId.copy()
+ } else {
+ lastMatchedRowId = null
+ }
+
+ if (emitNotMatchedTargetRows && !isSourceRowPresent) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresent) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ val mapFunc: InternalRow => InternalRow = if (performCardinalityCheck) {
Review comment:
Renamed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782367686
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Nah, it's fine as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775013059
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
Review comment:
I could add a rule to throw a descriptive error if this node stays after the analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775013059
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
Review comment:
I can add a rule to throw a descriptive error if this node stays after the analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776888877
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
Review comment:
Minor: I'd prefer to continue using `AppendData.byPosition`. I think that's easier to read when creating plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776886514
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
+
+ private lazy val analyzer: Analyzer = spark.sessionState.analyzer
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ case m @ UnresolvedMergeIntoIcebergTable(targetTable, sourceTable, context)
+ if targetTable.resolved && sourceTable.resolved && m.duplicateResolved =>
+
+ val resolvedMatchedActions = context.matchedActions.map {
+ case DeleteAction(cond) =>
+ val resolvedCond = cond.map(resolveCond("DELETE", _, m))
+ DeleteAction(resolvedCond)
+
+ case UpdateAction(cond, assignments) =>
+ val resolvedCond = cond.map(resolveCond("UPDATE", _, m))
+ // the update action can access columns from both target and source tables
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = false)
+ UpdateAction(resolvedCond, resolvedAssignments)
+
+ case UpdateStarAction(updateCondition) =>
+ val resolvedUpdateCondition = updateCondition.map(resolveCond("UPDATE", _, m))
+ val assignments = targetTable.output.map { attr =>
+ Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
+ }
+ // for UPDATE *, the value must be from the source table
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
+ UpdateAction(resolvedUpdateCondition, resolvedAssignments)
+
+ case o => o
Review comment:
This matches what Spark does, but since we use our own plan nodes we may want to fail sooner. I'm guessing that with our own merge node, we'd get a generic "not resolved" message instead of throwing a better one here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782360390
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
Review comment:
Yeah, I think it is not possible to hit this case as the parser would complain. Just added to be safe.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r785095472
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -280,29 +279,30 @@ public void testMergeWithSourceFromSetOps() {
}
@Test
- public void testMergeWithMultipleUpdatesForTargetRow() {
+ public void testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSource() {
Review comment:
Looks good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r783447452
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Sounds great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776890320
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
+
+ val nullabilityMap = attrs.indices.map { index =>
+ index -> outputs.exists(output => output(index).nullable)
+ }.toMap
+
+ val output = attrs.zipWithIndex.map { case (attr, index) =>
Review comment:
Can you add a comment explaining what's happening here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784124230
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
I've restructured this place a bit and added that comment right next to it. Let me know if that's better.
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784124517
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
+ }
+
+ private def startActionsResolved: Boolean = {
Review comment:
No longer applies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r781567963
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
Ahh I had overlooked the `if m.resolved` check. Thanks for pointing it out!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r780561731
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
There won't be an `InsertStarAction` or `UpdateStarAction` because `m.resolved` must be true and the plan is not resolved if there are any star actions: https://github.com/apache/iceberg/pull/3804/files#diff-aca4581053978b4b0146677000184c3200de1c0ed13505e58846659ece456538R37-R47
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775010568
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
Review comment:
I considered doing a hack where we could construct Spark `MergeIntoTable` and pass it through the original resolution rule and then convert back. The problem is that we cannot throw a descriptive error message when things don't work out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775013204
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
Review comment:
I am going to add tests for different join types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775011941
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/parser/extensions/IcebergSparkSqlExtensionsParser.scala
##########
@@ -127,13 +130,19 @@ class IcebergSparkSqlExtensionsParser(delegate: ParserInterface) extends ParserI
}
}
- private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan match {
+ private def replaceRowLevelCommands(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsDown {
Review comment:
I had to make it recursive to support CTE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776894718
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
+ val currentRowId = rowIdProj.apply(inputRow)
+ if (currentRowId == lastMatchedRowId) {
+ throw new SparkException(
+ "The ON search condition of the MERGE statement matched a single row from " +
+ "the target table with multiple rows of the source table. This could result " +
+ "in the target row being operated on more than once with an update or delete " +
+ "operation and is not allowed.")
+ }
+ lastMatchedRowId = currentRowId.copy()
+ } else {
+ lastMatchedRowId = null
+ }
+
+ if (emitNotMatchedTargetRows && !isSourceRowPresent) {
Review comment:
Is it expensive to call `processRow` here? I'd prefer not to duplicate it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r780550300
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
Do we need to handle `UpdateStarAction`?
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
+ case d: DeleteAction =>
+ d
+ case _ =>
+ throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
+ }
+
+ val alignedNotMatchedActions = m.notMatchedActions.map {
+ case i @ InsertAction(_, assignments) =>
Review comment:
Same question regarding `InsertStarAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782350137
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
+
+ private lazy val analyzer: Analyzer = spark.sessionState.analyzer
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ case m @ UnresolvedMergeIntoIcebergTable(targetTable, sourceTable, context)
+ if targetTable.resolved && sourceTable.resolved && m.duplicateResolved =>
+
+ val resolvedMatchedActions = context.matchedActions.map {
+ case DeleteAction(cond) =>
+ val resolvedCond = cond.map(resolveCond("DELETE", _, m))
+ DeleteAction(resolvedCond)
+
+ case UpdateAction(cond, assignments) =>
+ val resolvedCond = cond.map(resolveCond("UPDATE", _, m))
+ // the update action can access columns from both target and source tables
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = false)
+ UpdateAction(resolvedCond, resolvedAssignments)
+
+ case UpdateStarAction(updateCondition) =>
+ val resolvedUpdateCondition = updateCondition.map(resolveCond("UPDATE", _, m))
+ val assignments = targetTable.output.map { attr =>
+ Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
+ }
+ // for UPDATE *, the value must be from the source table
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
+ UpdateAction(resolvedUpdateCondition, resolvedAssignments)
+
+ case o => o
Review comment:
Yeah, we can throw an exception early here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782364585
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
Like construct a plan referring `child` and expressions in `MergeRowsParams`, @rdblue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784133534
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
No longer applies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784132649
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -280,29 +279,30 @@ public void testMergeWithSourceFromSetOps() {
}
@Test
- public void testMergeWithMultipleUpdatesForTargetRow() {
+ public void testMergeWithMultipleUpdatesForTargetRowSmallTargetLargeSource() {
Review comment:
@rdblue, this is the test we talked about. Right now, we are kind of safe as Spark broadcasts the left side of the join only if the join type is inner or right outer. In copy-on-write, it is either full outer or left outer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] Narcasserun commented on pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
Narcasserun commented on pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#issuecomment-1001463617
so Great
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776891720
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
Review comment:
Why is `rewriteMatchedActions` no longer needed in this version? Is that because this may not be optimized yet and the condition could be `1 = 1` instead of a `true` literal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776894139
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Should `childrenResolved` go first to avoid traversing expressions when the children prevent them from being resolved?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776893895
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
+ }
+
+ private def startActionsResolved: Boolean = {
Review comment:
Typo? I think you mean `star` not `start`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784128938
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData.byPosition(r, project)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, Nil))
+
+ // merge rows as there are multiple not matched actions
+ val mergeRows = MergeRows(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = Nil,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false,
+ output = buildMergeRowsOutput(Nil, notMatchedOutputs, r.output),
+ joinPlan)
+
+ AppendData.byPosition(r, mergeRows)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over
+ val matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral
+ val matchedOutputs = matchedActions.map(actionOutput(_, metadataAttrs)) :+ readAttrs
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, metadataAttrs))
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ val mergeRows = MergeRows(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = if (notMatchedActions.isEmpty) TrueLiteral else IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedConditions,
+ matchedOutputs = matchedOutputs,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = readAttrs,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true,
+ output = buildMergeRowsOutput(matchedOutputs, notMatchedOutputs, readAttrs),
+ joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def actionOutput(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Seq[Expression] = {
+
+ clause match {
+ case u: UpdateAction =>
+ u.assignments.map(_.value) ++ metadataAttrs
+
+ case _: DeleteAction =>
+ Nil
Review comment:
@rdblue, I had to replace `Seq[Option[Seq[Expression]]]` with `Seq[Seq[Expression]]` so that optimizer won't ignore the output expressions. That's why I had to switch to an empty output for `DeleteAction` instead of using `None`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] smaspe commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
smaspe commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r827249730
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,274 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData.byPosition(r, project)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val notMatchedConditions = notMatchedActions.map(actionCondition)
+ val notMatchedOutputs = notMatchedActions.map(actionOutput(_, Nil))
+
+ // merge rows as there are multiple not matched actions
+ val mergeRows = MergeRows(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedConditions,
+ notMatchedOutputs = notMatchedOutputs,
+ targetOutput = Nil,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false,
+ output = buildMergeRowsOutput(Nil, notMatchedOutputs, r.output),
+ joinPlan)
+
+ AppendData.byPosition(r, mergeRows)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
Review comment:
What are the unmatched destination rows needed for? It looks like it triggers a full scan on the destination table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784122361
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -310,39 +309,6 @@ public void testMergeWithMultipleUpdatesForTargetRow() {
sql("SELECT * FROM %s ORDER BY id ASC NULLS LAST", tableName));
}
- @Test
- public void testMergeWithDisabledCardinalityCheck() {
Review comment:
I'll deprecate it in a separate PR since this one is so huge.
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveMergeIntoTableReferences.scala
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.plans.logical.Assignment
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.InsertStarAction
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.catalyst.plans.logical.UpdateStarAction
+import org.apache.spark.sql.catalyst.rules.Rule
+
+/**
+ * A resolution rule similar to ResolveReferences in Spark but handles Iceberg MERGE operations.
+ */
+case class ResolveMergeIntoTableReferences(spark: SparkSession) extends Rule[LogicalPlan] {
+
+ private lazy val analyzer: Analyzer = spark.sessionState.analyzer
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ case m @ UnresolvedMergeIntoIcebergTable(targetTable, sourceTable, context)
+ if targetTable.resolved && sourceTable.resolved && m.duplicateResolved =>
+
+ val resolvedMatchedActions = context.matchedActions.map {
+ case DeleteAction(cond) =>
+ val resolvedCond = cond.map(resolveCond("DELETE", _, m))
+ DeleteAction(resolvedCond)
+
+ case UpdateAction(cond, assignments) =>
+ val resolvedCond = cond.map(resolveCond("UPDATE", _, m))
+ // the update action can access columns from both target and source tables
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = false)
+ UpdateAction(resolvedCond, resolvedAssignments)
+
+ case UpdateStarAction(updateCondition) =>
+ val resolvedUpdateCondition = updateCondition.map(resolveCond("UPDATE", _, m))
+ val assignments = targetTable.output.map { attr =>
+ Assignment(attr, UnresolvedAttribute(Seq(attr.name)))
+ }
+ // for UPDATE *, the value must be from the source table
+ val resolvedAssignments = resolveAssignments(assignments, m, resolveValuesWithSourceOnly = true)
+ UpdateAction(resolvedUpdateCondition, resolvedAssignments)
+
+ case o => o
Review comment:
Added an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784119844
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
I made all params as fields. That's seemed like the easiest option to get them optimized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784126788
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/MergeIntoIcebergTableResolutionCheck.scala
##########
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.UnresolvedMergeIntoIcebergTable
+
+object MergeIntoIcebergTableResolutionCheck extends (LogicalPlan => Unit) {
Review comment:
This is a check rule to run after the analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r780561731
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
There won't be an `InsertStarAction` or `UpdateStartAction` because `m.resolved` must be true and the plan is not resolved if there are any star actions: https://github.com/apache/iceberg/pull/3804/files#diff-aca4581053978b4b0146677000184c3200de1c0ed13505e58846659ece456538R37-R47
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
+ case d: DeleteAction =>
+ d
+ case _ =>
+ throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
+ }
+
+ val alignedNotMatchedActions = m.notMatchedActions.map {
+ case i @ InsertAction(_, assignments) =>
Review comment:
See above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
kbendick commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r780550300
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
Review comment:
Do we need to handle `UpdateStarAction`?
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/AlignRowLevelCommandAssignments.scala
##########
@@ -34,5 +41,73 @@ object AlignRowLevelCommandAssignments
override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
case u: UpdateIcebergTable if u.resolved && !u.aligned =>
u.copy(assignments = alignAssignments(u.table, u.assignments))
+
+ case m: MergeIntoIcebergTable if m.resolved && !m.aligned =>
+ val alignedMatchedActions = m.matchedActions.map {
+ case u @ UpdateAction(_, assignments) =>
+ u.copy(assignments = alignAssignments(m.targetTable, assignments))
+ case d: DeleteAction =>
+ d
+ case _ =>
+ throw new AnalysisException("Matched actions can only contain UPDATE or DELETE")
+ }
+
+ val alignedNotMatchedActions = m.notMatchedActions.map {
+ case i @ InsertAction(_, assignments) =>
Review comment:
Same question regarding `InsertStarAction`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r783362069
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
Review comment:
Actually, both `UpdateStarAction` and `InsertStarAction` are always unresolved. That's why this method is redundant. The actions are part of `expressions` and are being checked already. We can simply use the parent implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775016472
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
Review comment:
This is new and requires extra attention!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775014070
##########
File path: spark/v3.2/spark-extensions/src/test/java/org/apache/iceberg/spark/extensions/TestMerge.java
##########
@@ -595,6 +561,28 @@ public void testSelfMerge() {
assertEquals("Output should match", expectedRows, sql("SELECT * FROM %s ORDER BY id", tableName));
}
+ @Test
+ public void testSelfMergeWithCaching() {
Review comment:
We have 2 self merge tests already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776894603
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala
##########
@@ -0,0 +1,169 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.v2
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.BasePredicate
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.UnsafeProjection
+import org.apache.spark.sql.catalyst.expressions.codegen.GeneratePredicate
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.util.truncatedString
+import org.apache.spark.sql.execution.SparkPlan
+import org.apache.spark.sql.execution.UnaryExecNode
+
+case class MergeRowsExec(
+ params: MergeRowsParams,
+ output: Seq[Attribute],
+ child: SparkPlan) extends UnaryExecNode {
+
+ @transient override lazy val producedAttributes: AttributeSet = {
+ AttributeSet(output.filterNot(attr => inputSet.contains(attr)))
+ }
+
+ @transient override lazy val references: AttributeSet = child.outputSet
+
+ override def simpleString(maxFields: Int): String = {
+ s"MergeRowsExec${truncatedString(output, "[", ", ", "]", maxFields)}"
+ }
+
+ override protected def withNewChildInternal(newChild: SparkPlan): SparkPlan = {
+ copy(child = newChild)
+ }
+
+ protected override def doExecute(): RDD[InternalRow] = {
+ child.execute().mapPartitions {
+ processPartition(params, _)
+ }
+ }
+
+ private def createProjection(exprs: Seq[Expression], attrs: Seq[Attribute]): UnsafeProjection = {
+ UnsafeProjection.create(exprs, attrs)
+ }
+
+ private def createPredicate(expr: Expression, attrs: Seq[Attribute]): BasePredicate = {
+ GeneratePredicate.generate(expr, attrs)
+ }
+
+ private def applyProjection(
+ actions: Seq[(BasePredicate, Option[UnsafeProjection])],
+ inputRow: InternalRow): InternalRow = {
+
+ // find the first action where the predicate evaluates to true
+ // if there are overlapping conditions in actions, use the first matching action
+ // in the example below, when id = 5, both actions match but the first one is applied
+ // WHEN MATCHED AND id > 1 AND id < 10 UPDATE *
+ // WHEN MATCHED AND id = 5 OR id = 21 DELETE
+
+ val pair = actions.find {
+ case (predicate, _) => predicate.eval(inputRow)
+ }
+
+ // apply the projection to produce an output row, or return null to suppress this row
+ pair match {
+ case Some((_, Some(projection))) =>
+ projection.apply(inputRow)
+ case _ =>
+ null
+ }
+ }
+
+ private def processPartition(
+ params: MergeRowsParams,
+ rowIterator: Iterator[InternalRow]): Iterator[InternalRow] = {
+
+ val joinedAttrs = params.joinedAttributes
+ val isSourceRowPresentPred = createPredicate(params.isSourceRowPresent, joinedAttrs)
+ val isTargetRowPresentPred = createPredicate(params.isTargetRowPresent, joinedAttrs)
+ val matchedPreds = params.matchedConditions.map(createPredicate(_, joinedAttrs))
+ val matchedProjs = params.matchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val notMatchedPreds = params.notMatchedConditions.map(createPredicate(_, joinedAttrs))
+ val notMatchedProjs = params.notMatchedOutputs.map(_.map(createProjection(_, joinedAttrs)))
+ val projectTargetCols = createProjection(params.targetOutput, joinedAttrs)
+ val nonMatchedPairs = notMatchedPreds zip notMatchedProjs
+ val matchedPairs = matchedPreds zip matchedProjs
+ val rowIdAttrs = params.rowIdAttrs
+ val rowIdProj = createProjection(rowIdAttrs, joinedAttrs)
+ val performCardinalityCheck = params.performCardinalityCheck
+ val emitNotMatchedTargetRows = params.emitNotMatchedTargetRows
+
+ // This method is responsible for processing a input row to emit the resultant row with an
+ // additional column that indicates whether the row is going to be included in the final
+ // output of merge or not.
+ // 1. Found a target row for which there is no corresponding source row (join condition not met)
+ // - Only project the target columns if we need to output unchanged rows
+ // 2. Found a source row for which there is no corresponding target row (join condition not met)
+ // - Apply the not matched actions (i.e INSERT actions) if non match conditions are met.
+ // 3. Found a source row for which there is a corresponding target row (join condition met)
+ // - Apply the matched actions (i.e DELETE or UPDATE actions) if match conditions are met.
+ def processRow(inputRow: InternalRow): InternalRow = {
+ if (emitNotMatchedTargetRows && !isSourceRowPresentPred.eval(inputRow)) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresentPred.eval(inputRow)) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ var lastMatchedRowId: InternalRow = null
+
+ def processRowWithCardinalityCheck(inputRow: InternalRow): InternalRow = {
+ val isSourceRowPresent = isSourceRowPresentPred.eval(inputRow)
+ val isTargetRowPresent = isTargetRowPresentPred.eval(inputRow)
+
+ if (isSourceRowPresent && isTargetRowPresent) {
+ val currentRowId = rowIdProj.apply(inputRow)
+ if (currentRowId == lastMatchedRowId) {
+ throw new SparkException(
+ "The ON search condition of the MERGE statement matched a single row from " +
+ "the target table with multiple rows of the source table. This could result " +
+ "in the target row being operated on more than once with an update or delete " +
+ "operation and is not allowed.")
+ }
+ lastMatchedRowId = currentRowId.copy()
+ } else {
+ lastMatchedRowId = null
+ }
+
+ if (emitNotMatchedTargetRows && !isSourceRowPresent) {
+ projectTargetCols.apply(inputRow)
+ } else if (!isTargetRowPresent) {
+ applyProjection(nonMatchedPairs, inputRow)
+ } else {
+ applyProjection(matchedPairs, inputRow)
+ }
+ }
+
+ val mapFunc: InternalRow => InternalRow = if (performCardinalityCheck) {
Review comment:
Is there a better name than `mapFunc`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776893793
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
Review comment:
Good idea. We could just run a check in the optimizer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782364585
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
Like construct a plan referring to `child` and expressions in `MergeRowsParams`, @rdblue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r782365087
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
Review comment:
Will add then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784123157
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
Review comment:
I don't know why I changed that. Using `byPosition` in both places now.
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
Review comment:
No longer applies.
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
+
+ val nullabilityMap = attrs.indices.map { index =>
+ index -> outputs.exists(output => output(index).nullable)
+ }.toMap
+
+ val output = attrs.zipWithIndex.map { case (attr, index) =>
Review comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r784120204
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
Review comment:
Added a check rule that would run after the analysis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775012872
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/UnresolvedMergeIntoIcebergTable.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+/**
+ * A node that hides the MERGE condition and actions from regular Spark resolution.
+ */
+case class UnresolvedMergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ context: MergeIntoContext) extends BinaryCommand {
+
+ def duplicateResolved: Boolean = targetTable.outputSet.intersect(sourceTable.outputSet).isEmpty
Review comment:
I tested a number of self merge scenarios but could not trigger a case when the attrs would have to be deduplicated. That's why I did not add a separate rule for this. I checked the table resolution and relation cache access. Each time Spark looks up a relation in the cache, Spark clones it and assigns fresh expression IDs.
I'd appreciate if someone could check my understanding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] aokolnychyi commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
aokolnychyi commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r775016007
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeIntoIcebergTable.scala
##########
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.AssignmentUtils
+import org.apache.spark.sql.catalyst.expressions.Expression
+
+case class MergeIntoIcebergTable(
+ targetTable: LogicalPlan,
+ sourceTable: LogicalPlan,
+ mergeCondition: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction],
+ rewritePlan: Option[LogicalPlan] = None) extends RowLevelCommand {
+
+ override lazy val resolved: Boolean = {
+ expressions.forall(_.resolved) && childrenResolved && startActionsResolved
+ }
+
+ private def startActionsResolved: Boolean = {
+ val matchedActionsResolved = matchedActions.forall {
+ case _: UpdateStarAction => false
Review comment:
These are rewritten by `ResolveMergeIntoTableReferences` into regular `UpdateAction`s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776892579
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
Review comment:
If `notMatchedActions.isEmpty` then this can be `TrueLiteral`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776892897
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
Review comment:
This refactor looks correct to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776889805
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.Alias
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeReference
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.expressions.ExtendedV2ExpressionUtils
+import org.apache.spark.sql.catalyst.expressions.IsNotNull
+import org.apache.spark.sql.catalyst.expressions.Literal
+import org.apache.spark.sql.catalyst.expressions.Literal.FalseLiteral
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.expressions.MonotonicallyIncreasingID
+import org.apache.spark.sql.catalyst.plans.FullOuter
+import org.apache.spark.sql.catalyst.plans.LeftAnti
+import org.apache.spark.sql.catalyst.plans.LeftOuter
+import org.apache.spark.sql.catalyst.plans.logical.AppendData
+import org.apache.spark.sql.catalyst.plans.logical.DeleteAction
+import org.apache.spark.sql.catalyst.plans.logical.Filter
+import org.apache.spark.sql.catalyst.plans.logical.HintInfo
+import org.apache.spark.sql.catalyst.plans.logical.InsertAction
+import org.apache.spark.sql.catalyst.plans.logical.Join
+import org.apache.spark.sql.catalyst.plans.logical.JoinHint
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.MergeAction
+import org.apache.spark.sql.catalyst.plans.logical.MergeIntoIcebergTable
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRowsParams
+import org.apache.spark.sql.catalyst.plans.logical.NO_BROADCAST_HASH
+import org.apache.spark.sql.catalyst.plans.logical.Project
+import org.apache.spark.sql.catalyst.plans.logical.ReplaceData
+import org.apache.spark.sql.catalyst.plans.logical.UpdateAction
+import org.apache.spark.sql.connector.expressions.FieldReference
+import org.apache.spark.sql.connector.iceberg.catalog.SupportsRowLevelOperations
+import org.apache.spark.sql.connector.iceberg.write.RowLevelOperation.Command.MERGE
+import org.apache.spark.sql.connector.iceberg.write.SupportsDelta
+import org.apache.spark.sql.connector.write.RowLevelOperationTable
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+
+/**
+ * Assigns a rewrite plan for v2 tables that support rewriting data to handle MERGE statements.
+ *
+ * This rule assumes the commands have been fully resolved and all assignments have been aligned.
+ * That's why it must be run after AlignRowLevelCommandAssignments.
+ */
+object RewriteMergeIntoTable extends RewriteRowLevelCommand {
+
+ private final val ROW_FROM_SOURCE = "__row_from_source"
+ private final val ROW_FROM_TARGET = "__row_from_target"
+ private final val ROW_ID = "__row_id"
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators {
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty && notMatchedActions.size == 1 =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+ // NOT MATCHED conditions may only refer to columns in source so they can be pushed down
+ val insertAction = notMatchedActions.head.asInstanceOf[InsertAction]
+ val filteredSource = insertAction.condition match {
+ case Some(insertCond) => Filter(insertCond, source)
+ case None => source
+ }
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match the condition are appended to the table
+ val joinPlan = Join(filteredSource, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ val outputExprs = insertAction.assignments.map(_.value)
+ val outputColNames = r.output.map(_.name)
+ val outputCols = outputExprs.zip(outputColNames).map { case (expr, name) =>
+ Alias(expr, name)()
+ }
+ val project = Project(outputCols, joinPlan)
+
+ AppendData(r, project, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned && matchedActions.isEmpty =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r: DataSourceV2Relation =>
+
+ // when there are no MATCHED actions, use a left anti join to remove any matching rows
+ // and switch to using a regular append instead of a row-level merge
+ // only unmatched source rows that match action conditions are appended to the table
+ val joinPlan = Join(source, r, LeftAnti, Some(cond), JoinHint.NONE)
+
+ // we still have to merge rows as we have multiple not matched actions
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = TrueLiteral,
+ isTargetRowPresent = FalseLiteral,
+ matchedConditions = Nil,
+ matchedOutputs = Nil,
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, Nil)),
+ targetOutput = Nil,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Nil,
+ performCardinalityCheck = false,
+ emitNotMatchedTargetRows = false)
+ val mergeRows = buildMergeRows(mergeRowsParams, r.output, joinPlan)
+
+ AppendData(r, mergeRows, Map.empty, isByName = false)
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+
+ case m @ MergeIntoIcebergTable(aliasedTable, source, cond, matchedActions, notMatchedActions, None)
+ if m.resolved && m.aligned =>
+
+ EliminateSubqueryAliases(aliasedTable) match {
+ case r @ DataSourceV2Relation(tbl: SupportsRowLevelOperations, _, _, _, _) =>
+ val operation = buildRowLevelOperation(tbl, MERGE)
+ val table = RowLevelOperationTable(tbl, operation)
+ val rewritePlan = operation match {
+ case _: SupportsDelta =>
+ throw new AnalysisException("Delta merges are not currently supported")
+ case _ =>
+ buildReplaceDataPlan(r, table, source, cond, matchedActions, notMatchedActions)
+ }
+
+ m.copy(rewritePlan = Some(rewritePlan))
+
+ case p =>
+ throw new AnalysisException(s"$p is not an Iceberg table")
+ }
+ }
+
+ // build a rewrite plan for sources that support replacing groups of data (e.g. files, partitions)
+ private def buildReplaceDataPlan(
+ relation: DataSourceV2Relation,
+ table: RowLevelOperationTable,
+ source: LogicalPlan,
+ cond: Expression,
+ matchedActions: Seq[MergeAction],
+ notMatchedActions: Seq[MergeAction]): ReplaceData = {
+
+ // resolve all needed attrs (e.g. metadata attrs for grouping data on write)
+ val metadataAttrs = resolveRequiredMetadataAttrs(relation, table.operation)
+
+ // construct a scan relation and include all required metadata columns
+ val readRelation = buildReadRelation(relation, table, metadataAttrs)
+ val readAttrs = readRelation.output
+
+ // project an extra column to check if a target row exists after the join
+ // project a synthetic row ID to perform the cardinality check
+ val rowFromTarget = Alias(TrueLiteral, ROW_FROM_TARGET)()
+ val rowId = Alias(MonotonicallyIncreasingID(), ROW_ID)()
+ val targetTableProjExprs = readAttrs ++ Seq(rowFromTarget, rowId)
+ val targetTableProj = Project(targetTableProjExprs, readRelation)
+
+ // project an extra column to check if a source row exists after the join
+ val rowFromSource = Alias(TrueLiteral, ROW_FROM_SOURCE)()
+ val sourceTableProjExprs = source.output :+ rowFromSource
+ val sourceTableProj = Project(sourceTableProjExprs, source)
+
+ // use left outer join if there is no NOT MATCHED action, unmatched source rows can be discarded
+ // use full outer join in all other cases, unmatched source rows may be needed
+ // disable broadcasts for the target table to perform the cardinality check
+ val joinType = if (notMatchedActions.isEmpty) LeftOuter else FullOuter
+ val joinHint = JoinHint(leftHint = Some(HintInfo(Some(NO_BROADCAST_HASH))), rightHint = None)
+ val joinPlan = Join(targetTableProj, sourceTableProj, joinType, Some(cond), joinHint)
+
+ val rowIdAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_ID),
+ joinPlan)
+ val rowFromSourceAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_SOURCE),
+ joinPlan)
+ val rowFromTargetAttr = ExtendedV2ExpressionUtils.resolveRef[AttributeReference](
+ FieldReference(ROW_FROM_TARGET),
+ joinPlan)
+
+ // add an extra matched action to output the original row if none of the actual actions matched
+ // this is needed to keep target rows that should be copied over as we are working with groups
+ val mergeRowsParams = MergeRowsParams(
+ isSourceRowPresent = IsNotNull(rowFromSourceAttr),
+ isTargetRowPresent = IsNotNull(rowFromTargetAttr),
+ matchedConditions = matchedActions.map(actionCondition) :+ TrueLiteral,
+ matchedOutputs = matchedActions.map(output(_, metadataAttrs)) :+ Some(readAttrs),
+ notMatchedConditions = notMatchedActions.map(actionCondition),
+ notMatchedOutputs = notMatchedActions.map(output(_, metadataAttrs)),
+ targetOutput = readAttrs,
+ joinedAttributes = joinPlan.output,
+ rowIdAttrs = Seq(rowIdAttr),
+ performCardinalityCheck = isCardinalityCheckNeeded(matchedActions),
+ emitNotMatchedTargetRows = true)
+ val mergeRows = buildMergeRows(mergeRowsParams, readAttrs, joinPlan)
+
+ // build a plan to replace read groups in the table
+ val writeRelation = relation.copy(table = table)
+ ReplaceData(writeRelation, mergeRows, relation)
+ }
+
+ private def actionCondition(action: MergeAction): Expression = {
+ action.condition.getOrElse(TrueLiteral)
+ }
+
+ private def output(
+ clause: MergeAction,
+ metadataAttrs: Seq[Attribute]): Option[Seq[Expression]] = {
+
+ clause match {
+ case u: UpdateAction =>
+ Some(u.assignments.map(_.value) ++ metadataAttrs)
+
+ case _: DeleteAction =>
+ None
+
+ case i: InsertAction =>
+ Some(i.assignments.map(_.value) ++ metadataAttrs.map(attr => Literal(null, attr.dataType)))
+
+ case other =>
+ throw new AnalysisException(s"Unexpected action: $other")
+ }
+ }
+
+ private def buildMergeRows(
+ params: MergeRowsParams,
+ attrs: Seq[Attribute],
+ joinPlan: LogicalPlan): MergeRows = {
+
+ val outputs = params.matchedOutputs.flatten ++ params.notMatchedOutputs.flatten
+ assert(outputs.nonEmpty, "must be at least one output")
Review comment:
Should this be an analysis error instead? Or is there no valid SQL that could produce this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #3804: Spark: Implement copy-on-write MERGE
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #3804:
URL: https://github.com/apache/iceberg/pull/3804#discussion_r776893694
##########
File path: spark/v3.2/spark-extensions/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/MergeRows.scala
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.catalyst.plans.logical
+
+import org.apache.spark.sql.catalyst.expressions.Attribute
+import org.apache.spark.sql.catalyst.expressions.AttributeSet
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.catalyst.util.truncatedString
+
+// TODO: the optimizer currently ignores MergeRowsParams
Review comment:
Run the optimizer on each value and condition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org