[GitHub] [iceberg] aokolnychyi commented on a change in pull request #2116: Optimize join types for MERGE INTO

[GitBox <gi...@apache.org>]

aokolnychyi commented on a change in pull request #2116:
URL: https://github.com/apache/iceberg/pull/2116#discussion_r560544727



##########
File path: spark3-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteMergeInto.scala
##########
@@ -58,53 +61,138 @@ case class RewriteMergeInto(conf: SQLConf) extends Rule[LogicalPlan] with Rewrit
 
   override def apply(plan: LogicalPlan): LogicalPlan = {
     plan resolveOperators {
+      case MergeIntoTable(target: DataSourceV2Relation, source: LogicalPlan, cond, matchedActions, notMatchedActions)

Review comment:
       nit: do we need annotate `source` type explicitly?

##########
File path: spark3-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteMergeInto.scala
##########
@@ -58,53 +61,138 @@ case class RewriteMergeInto(conf: SQLConf) extends Rule[LogicalPlan] with Rewrit
 
   override def apply(plan: LogicalPlan): LogicalPlan = {
     plan resolveOperators {
+      case MergeIntoTable(target: DataSourceV2Relation, source: LogicalPlan, cond, matchedActions, notMatchedActions)
+          if matchedActions.isEmpty =>
+
+        val mergeBuilder = target.table.asMergeable.newMergeBuilder("merge", newWriteInfo(target.schema))
+        val targetTableScan = buildSimpleScanPlan(target.table, target.output, mergeBuilder, cond)
+
+        // when there are no matched actions, use a left anti join to remove any matching rows and rewrite to use
+        // append instead of replace. only unmatched source rows are passed to the merge and actions are all inserts.
+        val joinPlan = Join(source, targetTableScan, LeftAnti, Some(cond), JoinHint.NONE)
+
+        val mergeParams = MergeIntoParams(

Review comment:
       Is there a particular reason to use `MergeInto` node here? I think a simple projection will be enough and it will support codegen.

##########
File path: spark3-extensions/src/main/scala/org/apache/spark/sql/catalyst/optimizer/RewriteMergeInto.scala
##########
@@ -58,53 +61,138 @@ case class RewriteMergeInto(conf: SQLConf) extends Rule[LogicalPlan] with Rewrit
 
   override def apply(plan: LogicalPlan): LogicalPlan = {
     plan resolveOperators {
+      case MergeIntoTable(target: DataSourceV2Relation, source: LogicalPlan, cond, matchedActions, notMatchedActions)
+          if matchedActions.isEmpty =>
+
+        val mergeBuilder = target.table.asMergeable.newMergeBuilder("merge", newWriteInfo(target.schema))
+        val targetTableScan = buildSimpleScanPlan(target.table, target.output, mergeBuilder, cond)
+
+        // when there are no matched actions, use a left anti join to remove any matching rows and rewrite to use
+        // append instead of replace. only unmatched source rows are passed to the merge and actions are all inserts.
+        val joinPlan = Join(source, targetTableScan, LeftAnti, Some(cond), JoinHint.NONE)

Review comment:
       I think we should filter the source before the join if there is a condition in the insert action.
   
   ```
   ...
   WHEN NOT MATCHED AND inserCond
     THEN INSERT *
   ```
   
   Since we run in the optimizer, I am not sure whether Spark will do this automatically for us. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org