You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "karuppayya (via GitHub)" <gi...@apache.org> on 2025/09/21 01:55:38 UTC

[PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

karuppayya opened a new pull request, #52399:
URL: https://github.com/apache/spark/pull/52399

   ### What changes were proposed in this pull request?
   Generate code for MergeRowsExec operator
   
   ### Why are the changes needed?
   This would help operator fusing and improve performance of `MERGE` operations
   
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### How was this patch tested?
   All existing MergeRowsExec tests pass with codegen
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   No


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3806573426

   @juliuszsompolski @tomvanbussel @gengliangwang @viirya, can you folks review this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r2753899887


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        // Generate metric updates based on context
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return; // Discar row
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        // Codegen not implemented
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }
+  }
+
+  /**
+   * metric update code based on Keep's context
+   */
+  private def generateMetricUpdateCode(ctx: CodegenContext, context: Context,
+                                       sourcePresent: Boolean): String = {
+    context match {
+      case Copy =>
+        val copyMetric = metricTerm(ctx, "numTargetRowsCopied")
+        s"$copyMetric.add(1);"
+
+      case Insert =>
+        val insertMetric = metricTerm(ctx, "numTargetRowsInserted")
+        s"$insertMetric.add(1);"
+
+      case Update =>
+        generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+      case Delete =>
+        generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+      case _ =>
+        throw new IllegalArgumentException(s"Unexpected context for KeepExec: $context")
+    }
+  }
+
+  private def generateUpdateMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val updateMetric = metricTerm(ctx, "numTargetRowsUpdated")
+    if (sourcePresent) {
+      val matchedUpdateMetric = metricTerm(ctx, "numTargetRowsMatchedUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$matchedUpdateMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceUpdateMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$notMatchedBySourceUpdateMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  private def generateDeleteMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val deleteMetric = metricTerm(ctx, "numTargetRowsDeleted")
+    if (sourcePresent) {
+      val matchedDeleteMetric = metricTerm(ctx, "numTargetRowsMatchedDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$matchedDeleteMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceDeleteMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$notMatchedBySourceDeleteMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  /**
+   * Helper method to save and restore CodegenContext state for code generation.
+   *
+   * This is needed because when generating code for expressions, the CodegenContext
+   * state (currentVars and INPUT_ROW) gets modified during expression evaluation.
+   * This method temporarily sets the context to the input variables from doConsume
+   * and restores the original state after the block completes.
+   */
+  private def withCodegenContext[T](
+      ctx: CodegenContext,
+      inputCurrentVars: Seq[ExprCode])(block: => T): T = {
+    val originalCurrentVars = ctx.currentVars
+    val originalInputRow = ctx.INPUT_ROW
+    try {
+      // Set to the input variables saved in doConsume
+      ctx.currentVars = inputCurrentVars
+      block
+    } finally {
+      // Restore original context
+      ctx.currentVars = originalCurrentVars
+      ctx.INPUT_ROW = originalInputRow
+    }
+  }
+
+  private def generatePredicateCode(ctx: CodegenContext,
+                                    predicate: Expression,
+                                    inputAttrs: Seq[Attribute],
+                                    inputCurrentVars: Seq[ExprCode]): ExprCode = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundPredicate = BindReferences.bindReference(predicate, inputAttrs)
+      val ev = boundPredicate.genCode(ctx)
+      val predicateVar = ctx.freshName("predicateResult")
+      val code = code"""
+                       |${ev.code}
+                       |boolean $predicateVar = !${ev.isNull} && ${ev.value};
+                        """.stripMargin
+      ExprCode(code, FalseLiteral,
+        JavaCode.variable(predicateVar, BooleanType))
+    }
+  }
+
+  private def generateProjectionCode(ctx: CodegenContext,
+                                     outputExprs: Seq[Expression],
+                                     inputCurrentVars: Seq[ExprCode]): Seq[ExprCode] = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundExprs = outputExprs.map(BindReferences.bindReference(_, child.output))
+      boundExprs.map(_.genCode(ctx))
+    }
+  }

Review Comment:
   this code is quite generic. Aren't there existing helpers for this? If not, maybe move it to some generic helper (it's stateless, so companion `object CodegenSupport`)?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,

Review Comment:
   parameter instructionType is not used.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        // Generate metric updates based on context
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return; // Discar row
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        // Codegen not implemented
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }
+  }
+
+  /**
+   * metric update code based on Keep's context
+   */
+  private def generateMetricUpdateCode(ctx: CodegenContext, context: Context,
+                                       sourcePresent: Boolean): String = {
+    context match {
+      case Copy =>
+        val copyMetric = metricTerm(ctx, "numTargetRowsCopied")
+        s"$copyMetric.add(1);"
+
+      case Insert =>
+        val insertMetric = metricTerm(ctx, "numTargetRowsInserted")
+        s"$insertMetric.add(1);"
+
+      case Update =>
+        generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+      case Delete =>
+        generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+      case _ =>
+        throw new IllegalArgumentException(s"Unexpected context for KeepExec: $context")
+    }
+  }

Review Comment:
   move it to Instruction class.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {

Review Comment:
   scalastyle: break one parameter per line, 4 space indent for parameters (here and in other places)
   ```suggestion
     private def generateInstructionsCode(
         ctx: CodegenContext,
         instructions: Seq[Instruction],
         instructionType: String,
         inputExprs: Seq[ExprCode],
         sourcePresent: Boolean): String = {
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {

Review Comment:
   for better code locality and readability, move this method to the Instruction class, with each subclass overriding it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3837597985

   Thanks for explaining. I will soon add the benchmark code and results as part of this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3107950240


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(

Review Comment:
   They are exactly same, move it to `CodegenSupport` for access across operators



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed pull request #52399: [SPARK-53652][SQL]Codegen for MergeRowExec
URL: https://github.com/apache/spark/pull/52399


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3108080174


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(

Review Comment:
    Changed its visibility from `private` to `protected`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3098564309


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return;  // Do nothing
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }

Review Comment:
   Interpreted path seems throwing `AnalysisException`? Shall we keep them consistent?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}

Review Comment:
   Would be better to add a comment here saying why no `else` branch is needed.



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;

Review Comment:
   Recommend to add comment:
   ```suggestion
            |return; // no instruction matched, discard row
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}

Review Comment:
   ```suggestion
              |// MergeRowsExec: Keep(${context}) instruction
              |${code.code}
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}

Review Comment:
   ```suggestion
          |// MergeRowsExec: evaluate source/target row presence
          |${sourcePresentExpr.code}
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}

Review Comment:
   ```suggestion
              |// MergeRowsExec: Discard instruction
              |${code.code}
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}

Review Comment:
   ```suggestion
               |// MergeRowsExec: cardinality validation (detect duplicate target matches)
               |${currentRowId.code}
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {

Review Comment:
   ```suggestion
          |// MergeRowsExec: route to matched/notMatched/notMatchedBySource instructions
          |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
   ```



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return;  // Do nothing
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}

Review Comment:
   ```suggestion
              |// MergeRowsExec: Split instruction
              |${code.code}
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3349359424

   @aokolnychyi @cloud-fan @dongjoon-hyun @rdblue Could one of you please review this PR?
   
   This change specifically improves[ MERGE INTO](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RewriteMergeIntoTable.scala) operations by fusing the Join Operator (typically SortMerge) and the MergeRow operator.
   
   This fusion is expected to significantly boost MERGE performance at scale by reducing virtual function call overhead. Furthermore, it enables this path to be code-generated and potentially interoperate with other codegen-supported operators(most operators in Spark) in the future.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4306910763

   Merged to master. Thanks @karuppayya @juliuszsompolski 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3076248849


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,288 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    generateInstructionExecutionCode(ctx, input)
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, inputExprs, sourcePresent = false)

Review Comment:
   Done



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(

Review Comment:
    logic similar to `org.apache.spark.sql.execution.CodegenSupport.constructConsumeParameters`



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,288 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    generateInstructionExecutionCode(ctx, input)
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;

Review Comment:
   Make sense, especially given that toggling configs(`spark.sql.codegen.splitConsumeFuncByOperator`) can easily trigger it
   ```
     val consumeFunc = if (confEnabled && requireAllOutput
         && CodeGenerator.isValidParamLength(paramLength)) {
       constructDoConsumeFunction(ctx, inputVars, row)  
     } else {
       parent.doConsume(ctx, inputVars, rowVar)
     }
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4283649883

   I will go to merge this once the benchmark files are added. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3104755558


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(

Review Comment:
   Is it possible to merge them if they are very similar?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,

Review Comment:
   Can you add a comment explaining why to wrap this into a separate function?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}

Review Comment:
   It would be good to have these comment in the generated Java code to help debugging.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3107950240


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(

Review Comment:
   They are exactly same, move it to `CodegenSupport` for access across operators



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,340 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  override def supportCodegen: Boolean = {
+    CodeGenerator.isValidParamLength(CodeGenerator.calculateParamLength(child.output))
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,

Review Comment:
   done



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}

Review Comment:
   Added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3809460302

   By providing the codegen path, it means existing tests related to MergeRowExec will be run under codegen path.
   It probably also means that we won't test the interpreted path anymore.
   It would be better if this can add tests specifically validating codegen and interpreted path correctness together.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4264168212

   @viirya @cloud-fan, could you please review too?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3098740081


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)

Review Comment:
   CodegenSupport will check if the parameters length is valid under JVM. Do we need to check it here too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3113622479


##########
sql/core/benchmarks/MergeRowsExecBenchmark-jdk21-results.txt:
##########
@@ -0,0 +1,54 @@
+================================================================================================
+MergeRowsExec Codegen Benchmark

Review Comment:
   Added remaining



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3068313181


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {

Review Comment:
   Fixed



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        // Generate metric updates based on context
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return; // Discar row
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        // Codegen not implemented
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }
+  }
+
+  /**
+   * metric update code based on Keep's context
+   */
+  private def generateMetricUpdateCode(ctx: CodegenContext, context: Context,
+                                       sourcePresent: Boolean): String = {
+    context match {
+      case Copy =>
+        val copyMetric = metricTerm(ctx, "numTargetRowsCopied")
+        s"$copyMetric.add(1);"
+
+      case Insert =>
+        val insertMetric = metricTerm(ctx, "numTargetRowsInserted")
+        s"$insertMetric.add(1);"
+
+      case Update =>
+        generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+      case Delete =>
+        generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+      case _ =>
+        throw new IllegalArgumentException(s"Unexpected context for KeepExec: $context")
+    }
+  }
+
+  private def generateUpdateMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val updateMetric = metricTerm(ctx, "numTargetRowsUpdated")
+    if (sourcePresent) {
+      val matchedUpdateMetric = metricTerm(ctx, "numTargetRowsMatchedUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$matchedUpdateMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceUpdateMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$notMatchedBySourceUpdateMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  private def generateDeleteMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val deleteMetric = metricTerm(ctx, "numTargetRowsDeleted")
+    if (sourcePresent) {
+      val matchedDeleteMetric = metricTerm(ctx, "numTargetRowsMatchedDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$matchedDeleteMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceDeleteMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$notMatchedBySourceDeleteMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  /**
+   * Helper method to save and restore CodegenContext state for code generation.
+   *
+   * This is needed because when generating code for expressions, the CodegenContext
+   * state (currentVars and INPUT_ROW) gets modified during expression evaluation.
+   * This method temporarily sets the context to the input variables from doConsume
+   * and restores the original state after the block completes.
+   */
+  private def withCodegenContext[T](
+      ctx: CodegenContext,
+      inputCurrentVars: Seq[ExprCode])(block: => T): T = {
+    val originalCurrentVars = ctx.currentVars
+    val originalInputRow = ctx.INPUT_ROW
+    try {
+      // Set to the input variables saved in doConsume
+      ctx.currentVars = inputCurrentVars
+      block
+    } finally {
+      // Restore original context
+      ctx.currentVars = originalCurrentVars
+      ctx.INPUT_ROW = originalInputRow
+    }
+  }
+
+  private def generatePredicateCode(ctx: CodegenContext,
+                                    predicate: Expression,
+                                    inputAttrs: Seq[Attribute],
+                                    inputCurrentVars: Seq[ExprCode]): ExprCode = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundPredicate = BindReferences.bindReference(predicate, inputAttrs)
+      val ev = boundPredicate.genCode(ctx)
+      val predicateVar = ctx.freshName("predicateResult")
+      val code = code"""
+                       |${ev.code}
+                       |boolean $predicateVar = !${ev.isNull} && ${ev.value};
+                        """.stripMargin
+      ExprCode(code, FalseLiteral,
+        JavaCode.variable(predicateVar, BooleanType))
+    }
+  }
+
+  private def generateProjectionCode(ctx: CodegenContext,
+                                     outputExprs: Seq[Expression],
+                                     inputCurrentVars: Seq[ExprCode]): Seq[ExprCode] = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundExprs = outputExprs.map(BindReferences.bindReference(_, child.output))
+      boundExprs.map(_.genCode(ctx))
+    }
+  }

Review Comment:
   The save/restore pattern in `withCodegenContext` is not being used by other codegen operators atm—  Since there's no other consumer for this pattern, I'll keep it private in MergeRowsExec for now and move it when a second use case appears? (The other methods depend on `withCodegenContext`). 



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {

Review Comment:
    This was not possible, since `org.apache.spark.sql.catalyst.plans.logical.MergeRows.Instruction` is in the `catalyst` module whereas the codegen code lives in `core`.
   Methods that are not referncible
   1. `org.apache.spark.sql.execution.CodegenSupport#consume`
   2. `org.apache.spark.sql.execution.CodegenSupport#metricTerm`



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,

Review Comment:
   Removed



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        // Generate metric updates based on context
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return; // Discar row
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        // Codegen not implemented
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }
+  }
+
+  /**
+   * metric update code based on Keep's context
+   */
+  private def generateMetricUpdateCode(ctx: CodegenContext, context: Context,
+                                       sourcePresent: Boolean): String = {
+    context match {
+      case Copy =>
+        val copyMetric = metricTerm(ctx, "numTargetRowsCopied")
+        s"$copyMetric.add(1);"
+
+      case Insert =>
+        val insertMetric = metricTerm(ctx, "numTargetRowsInserted")
+        s"$insertMetric.add(1);"
+
+      case Update =>
+        generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+      case Delete =>
+        generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+      case _ =>
+        throw new IllegalArgumentException(s"Unexpected context for KeepExec: $context")
+    }
+  }

Review Comment:
   [same](https://github.com/apache/spark/pull/52399/changes#r3068316198)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3315430297

   @aokolnychyi @rdblue @szehon-ho @cloud-fan @huaxingao @sunchao  Can you help review this?
   cc: @RussellSpitzer @anuragmantri


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3837531978

   @juliuszsompolski @viirya Thanks for your reviews, I will have them handled.
   I agree that we need to validate that the ionterpreted path and codegen path generate the same results.
   Like mentioned, I didnt see that a general pattern and didnot add it initially.
   
   I see this `org.apache.spark.sql.catalyst.expressions.aggregate.TestWithAndWithoutCodegen`,  under `expressions.aggregaate`.
   Should we move this at a top level in `sql/catalyst` or create a new one for use in `sql/core`?
   (I think we could reuse sql/catalyst by moving to top level)
   
   > Is there benchmark numbers showing actual performance improvements?
   
   @viirya This implementation follows the standard pattern of codegen-enabling operators to allow for **operator fusing** and to eliminate the **overhead of virtual function calls**. Since most Spark operators are codegen-enabled, it makes sense to have MergeRows follow suit.
   But if you feel strongly that we need empirical data to justify the implementation, I’m happy to add a benchmark file to verify the gains
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r2758435272


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,277 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    // Save the input variables that were passed to doConsume
+    val inputCurrentVars = input
+
+    // code for instruction execution code
+    generateInstructionExecutionCode(ctx, inputCurrentVars)
+  }
+
+
+  /**
+   * code for cardinality validation
+   */
+  private def generateCardinalityValidationCode(ctx: CodegenContext, rowIdOrdinal: Int,
+                                                input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(ctx: CodegenContext,
+                                               inputExprs: Seq[ExprCode]): String = {
+
+    // code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      "matched", inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      "notMatched", inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, "notMatchedBySource", inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(ctx: CodegenContext, instructions: Seq[Instruction],
+                                       instructionType: String,
+                                       inputExprs: Seq[ExprCode],
+                                       sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(ctx: CodegenContext,
+                                            instruction: Instruction,
+                                            inputExprs: Seq[ExprCode],
+                                            sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        // Generate metric updates based on context
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return; // Discar row
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        // Codegen not implemented
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }
+  }
+
+  /**
+   * metric update code based on Keep's context
+   */
+  private def generateMetricUpdateCode(ctx: CodegenContext, context: Context,
+                                       sourcePresent: Boolean): String = {
+    context match {
+      case Copy =>
+        val copyMetric = metricTerm(ctx, "numTargetRowsCopied")
+        s"$copyMetric.add(1);"
+
+      case Insert =>
+        val insertMetric = metricTerm(ctx, "numTargetRowsInserted")
+        s"$insertMetric.add(1);"
+
+      case Update =>
+        generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+      case Delete =>
+        generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+      case _ =>
+        throw new IllegalArgumentException(s"Unexpected context for KeepExec: $context")
+    }
+  }
+
+  private def generateUpdateMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val updateMetric = metricTerm(ctx, "numTargetRowsUpdated")
+    if (sourcePresent) {
+      val matchedUpdateMetric = metricTerm(ctx, "numTargetRowsMatchedUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$matchedUpdateMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceUpdateMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceUpdated")
+
+      s"""
+         |$updateMetric.add(1);
+         |$notMatchedBySourceUpdateMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  private def generateDeleteMetricUpdateCode(ctx: CodegenContext,
+                                             sourcePresent: Boolean): String = {
+    val deleteMetric = metricTerm(ctx, "numTargetRowsDeleted")
+    if (sourcePresent) {
+      val matchedDeleteMetric = metricTerm(ctx, "numTargetRowsMatchedDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$matchedDeleteMetric.add(1);
+       """.stripMargin
+    } else {
+      val notMatchedBySourceDeleteMetric = metricTerm(ctx, "numTargetRowsNotMatchedBySourceDeleted")
+
+      s"""
+         |$deleteMetric.add(1);
+         |$notMatchedBySourceDeleteMetric.add(1);
+       """.stripMargin
+    }
+  }
+
+  /**
+   * Helper method to save and restore CodegenContext state for code generation.
+   *
+   * This is needed because when generating code for expressions, the CodegenContext
+   * state (currentVars and INPUT_ROW) gets modified during expression evaluation.
+   * This method temporarily sets the context to the input variables from doConsume
+   * and restores the original state after the block completes.
+   */
+  private def withCodegenContext[T](
+      ctx: CodegenContext,
+      inputCurrentVars: Seq[ExprCode])(block: => T): T = {
+    val originalCurrentVars = ctx.currentVars
+    val originalInputRow = ctx.INPUT_ROW
+    try {
+      // Set to the input variables saved in doConsume
+      ctx.currentVars = inputCurrentVars
+      block
+    } finally {
+      // Restore original context
+      ctx.currentVars = originalCurrentVars
+      ctx.INPUT_ROW = originalInputRow
+    }
+  }
+
+  private def generatePredicateCode(ctx: CodegenContext,
+                                    predicate: Expression,
+                                    inputAttrs: Seq[Attribute],
+                                    inputCurrentVars: Seq[ExprCode]): ExprCode = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundPredicate = BindReferences.bindReference(predicate, inputAttrs)
+      val ev = boundPredicate.genCode(ctx)
+      val predicateVar = ctx.freshName("predicateResult")
+      val code = code"""
+                       |${ev.code}
+                       |boolean $predicateVar = !${ev.isNull} && ${ev.value};
+                        """.stripMargin
+      ExprCode(code, FalseLiteral,
+        JavaCode.variable(predicateVar, BooleanType))
+    }
+  }
+
+  private def generateProjectionCode(ctx: CodegenContext,
+                                     outputExprs: Seq[Expression],
+                                     inputCurrentVars: Seq[ExprCode]): Seq[ExprCode] = {
+    withCodegenContext(ctx, inputCurrentVars) {
+      val boundExprs = outputExprs.map(BindReferences.bindReference(_, child.output))
+      boundExprs.map(_.genCode(ctx))
+    }
+  }

Review Comment:
   It looks like [object CodeGenerator](https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L1435C1-L1435C21) would be the best place to add generic helpers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "aokolnychyi (via GitHub)" <gi...@apache.org>.
aokolnychyi commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4264170544

   @tomvanbussel FYI if you are interested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3102791264


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)
+    val body = generateInstructionExecutionCode(ctx, paramExprs)
+
+    ctx.addNewFunction(funcName,
+      s"""
+         |private void $funcName(${params.mkString(", ")})
+         |    throws java.io.IOException {
+         |  $body
+         |}
+       """.stripMargin)
+
+    s"$funcName(${args.mkString(", ")});"
+  }
+
+  private def buildConsumeParameters(
+      ctx: CodegenContext,
+      input: Seq[ExprCode]): (Seq[String], Seq[String], Seq[ExprCode]) = {
+    val args = Seq.newBuilder[String]
+    val params = Seq.newBuilder[String]
+    val paramExprs = Seq.newBuilder[ExprCode]
+
+    input.zip(child.output).zipWithIndex.foreach {
+      case ((ev, attr), i) =>
+        val paramName = ctx.freshName(s"mergeExpr_$i")
+        val paramType = CodeGenerator.javaType(attr.dataType)
+        args += ev.value.toString
+        params += s"$paramType $paramName"
+
+        val paramIsNull = if (!attr.nullable) {
+          FalseLiteral
+        } else {
+          val isNull = ctx.freshName(s"mergeIsNull_$i")
+          args += ev.isNull.toString
+          params += s"boolean $isNull"
+          JavaCode.isNullVariable(isNull)
+        }
+
+        paramExprs += ExprCode(
+          paramIsNull,
+          JavaCode.variable(paramName, attr.dataType))
+    }
+
+    (args.result(), params.result(), paramExprs.result())
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(
+      ctx, matchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedInstructions, inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(
+      ctx, notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+
+      case Discard(condition) =>
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateDeleteMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  return;  // Do nothing
+           |}
+       """.stripMargin
+
+      case Split(condition, outputExprs, otherOutputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val otherProjectionExpr = generateProjectionCode(ctx, otherOutputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+        val metricUpdateCode = generateUpdateMetricUpdateCode(ctx, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  ${consume(ctx, otherProjectionExpr)}
+           |  return;
+           |}
+       """.stripMargin
+      case _ =>
+        throw new SparkUnsupportedOperationException(
+          errorClass = "_LEGACY_ERROR_TEMP_3073",
+          messageParameters = Map("instruction" -> instruction.toString))
+    }

Review Comment:
   `SparkUnsupportedOperationException` seems more suited than `AnalysisException`.
   @aokolnychyi what do you think? (Since you added [this](https://github.com/apache/spark/pull/41448/changes#diff-7d3bcef84fbd618500473fc8329c3bae2fe8ca7ff6bc5d6dbd66a9fbca1e2e43R341)). Should we change interpreted path?



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,331 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    val funcName = ctx.freshName("mergeProcessRow")
+    val (args, params, paramExprs) =
+      buildConsumeParameters(ctx, input)

Review Comment:
   I have overridden `org.apache.spark.sql.execution.CodegenSupport#supportCodegen` to check for `org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.isValidParamLength`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3498562801

   @cloud-fan @aokolnychyi  Would you be able to help review this? Or let me know who can help here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3113035753


##########
sql/core/benchmarks/MergeRowsExecBenchmark-jdk21-results.txt:
##########
@@ -0,0 +1,54 @@
+================================================================================================
+MergeRowsExec Codegen Benchmark

Review Comment:
   We are used to have three files for each benchmark. For consistency, can you add -jdk25-results.txt and -results.txt?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4296925022

   @karuppayya Can you re-trigger CI?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3834812294

   It would be great to add some dedicated tests that check in the plan that WSCG triggered.
   Like @viirya mentioned, we if it codegens, we would also now be testing with codegen and not with no-codegen, but this is a general problem of all Spark SQL... @viirya do we have other places that re-test SparkSQL execution with and without codegen enabled? We could add a duplicated test suite, one enabling and one disabling codegen, though I don't see it being done in other places...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4306856825

   @viirya done, CI green.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "karuppayya (via GitHub)" <gi...@apache.org>.
karuppayya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-4245524560

   Thanks @juliuszsompolski for the review.
   @viirya Can you also help review this chnage. Thank you


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "juliuszsompolski (via GitHub)" <gi...@apache.org>.
juliuszsompolski commented on code in PR #52399:
URL: https://github.com/apache/spark/pull/52399#discussion_r3072062597


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,288 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    generateInstructionExecutionCode(ctx, input)
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, inputExprs, sourcePresent = false)

Review Comment:
   style nit: don't break lines in the middle of parameter list.
   either
   ```
       val matchedInstructionsCode = 
         generateInstructionsCode(ctx, matchedInstructions, inputExprs, sourcePresent = true)
   ```
   or
   ```
       val matchedInstructionsCode = generateInstructionsCode(
         ctx, matchedInstructions, inputExprs, sourcePresent = true)
   ```
   or
   ```
       val matchedInstructionsCode = generateInstructionsCode(
         ctx,
         matchedInstructions,
         inputExprs,
         sourcePresent = true)
   ```
   



##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/MergeRowsExec.scala:
##########
@@ -92,6 +95,288 @@ case class MergeRowsExec(
     child.execute().mapPartitions(processPartition)
   }
 
+  override def inputRDDs(): Seq[RDD[InternalRow]] = {
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+  }
+
+  override def needCopyResult: Boolean = {
+    // Only Split instructions produce multiple output rows per input row.
+    val hasSplitInstruction = (matchedInstructions ++ notMatchedInstructions ++
+      notMatchedBySourceInstructions).exists(_.isInstanceOf[Split])
+    hasSplitInstruction ||
+      child.asInstanceOf[CodegenSupport].needCopyResult
+  }
+
+  protected override def doProduce(ctx: CodegenContext): String = {
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+  }
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: ExprCode): String = {
+    generateInstructionExecutionCode(ctx, input)
+  }
+
+  /**
+   * Code for cardinality validation.
+   */
+  private def generateCardinalityValidationCode(
+      ctx: CodegenContext,
+      rowIdOrdinal: Int,
+      input: Seq[ExprCode]): ExprCode = {
+    val bitmapClass = classOf[Roaring64Bitmap]
+    val rowIdBitmap = ctx.addMutableState(bitmapClass.getName, "matchedRowIds",
+      v => s"$v = new ${bitmapClass.getName}();")
+
+    val currentRowId = input(rowIdOrdinal)
+    val queryExecutionErrorsClass = QueryExecutionErrors.getClass.getName + ".MODULE$"
+    val code =
+      code"""
+            |${currentRowId.code}
+            |if ($rowIdBitmap.contains(${currentRowId.value})) {
+            |  throw $queryExecutionErrorsClass.mergeCardinalityViolationError();
+            |}
+            |$rowIdBitmap.add(${currentRowId.value});
+     """.stripMargin
+    ExprCode(code, FalseLiteral, JavaCode.variable(rowIdBitmap, bitmapClass))
+  }
+
+  /**
+   * Generate code for instruction execution based on row presence conditions
+   */
+  private def generateInstructionExecutionCode(
+      ctx: CodegenContext,
+      inputExprs: Seq[ExprCode]): String = {
+
+    // Code for evaluating src/tgt presence conditions
+    val sourcePresentExpr = generatePredicateCode(ctx, isSourceRowPresent, child.output, inputExprs)
+    val targetPresentExpr = generatePredicateCode(ctx, isTargetRowPresent, child.output, inputExprs)
+
+    // Code for each instruction type
+    val matchedInstructionsCode = generateInstructionsCode(ctx, matchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedInstructionsCode = generateInstructionsCode(ctx, notMatchedInstructions,
+      inputExprs, sourcePresent = true)
+    val notMatchedBySourceInstructionsCode = generateInstructionsCode(ctx,
+      notMatchedBySourceInstructions, inputExprs, sourcePresent = false)
+
+    val cardinalityValidationCode = if (checkCardinality) {
+      val rowIdOrdinal = child.output.indexWhere(attr => conf.resolver(attr.name, ROW_ID))
+      assert(rowIdOrdinal != -1, "Cannot find row ID attr")
+      generateCardinalityValidationCode(ctx, rowIdOrdinal, inputExprs).code
+    } else {
+      ""
+    }
+
+    s"""
+       |${sourcePresentExpr.code}
+       |${targetPresentExpr.code}
+       |
+       |if (${targetPresentExpr.value} && ${sourcePresentExpr.value}) {
+       |  $cardinalityValidationCode
+       |  $matchedInstructionsCode
+       |} else if (${sourcePresentExpr.value}) {
+       |  $notMatchedInstructionsCode
+       |} else if (${targetPresentExpr.value}) {
+       |  $notMatchedBySourceInstructionsCode
+       |}
+     """.stripMargin
+  }
+
+  /**
+   * Generate code for executing a sequence of instructions
+   */
+  private def generateInstructionsCode(
+      ctx: CodegenContext,
+      instructions: Seq[Instruction],
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    if (instructions.isEmpty) {
+      ""
+    } else {
+      val instructionCodes = instructions.map(instruction =>
+        generateSingleInstructionCode(ctx, instruction, inputExprs, sourcePresent))
+
+      s"""
+         |${instructionCodes.mkString("\n")}
+         |return;
+       """.stripMargin
+    }
+  }
+
+  private def generateSingleInstructionCode(
+      ctx: CodegenContext,
+      instruction: Instruction,
+      inputExprs: Seq[ExprCode],
+      sourcePresent: Boolean): String = {
+    instruction match {
+      case Keep(context, condition, outputExprs) =>
+        val projectionExpr = generateProjectionCode(ctx, outputExprs, inputExprs)
+        val code = generatePredicateCode(ctx, condition, child.output, inputExprs)
+
+        val metricUpdateCode = generateMetricUpdateCode(ctx, context, sourcePresent)
+
+        s"""
+           |${code.code}
+           |if (${code.value}) {
+           |  $metricUpdateCode
+           |  ${consume(ctx, projectionExpr)}
+           |  return;

Review Comment:
   These "returns" in here and other instructions gave me a bit of pause - I had to investigate why this wouldn't exit out of some outer produce loop prematurely.
   I investigated a bit with Claude, and it seems it could work, but I think it can be dangerous:
   
   ```
     ---
     The return in the generated Java code is safe because it doesn't live directly in processNext() — it lives inside a separate wrapper method created by constructDoConsumeFunction.
   
     Here's what happens. When MergeRowsExec's child (the scan) calls this.consume(ctx, vars), the consume() method at WholeStageCodegenExec.scala:196-201 decides how to incorporate MergeRowsExec's doConsume output:
   
     val consumeFunc = if (confEnabled && requireAllOutput
         && CodeGenerator.isValidParamLength(paramLength)) {
       constructDoConsumeFunction(ctx, inputVars, row)  // wrap in separate method
     } else {
       parent.doConsume(ctx, inputVars, rowVar)          // inline
     }
   
     When the (default-on) splitConsumeFuncByOperator config is active, constructDoConsumeFunction (line 213) wraps MergeRowsExec's entire doConsume output — including all instruction code with return; — inside a private Java method:
   
     private void doConsume_mergeRows(/* child output cols */) throws IOException {
       // sourcePresentExpr, targetPresentExpr
       if (targetPresent && sourcePresent) {
         if (keepPredicate) {
           metricUpdate;
           append(projectedRow);
           return;    // <-- exits doConsume_mergeRows, NOT processNext
         }
       }
       // ... other branches
     }
   
     And the produce loop just calls it:
   
     // in processNext():
     while (input.hasNext()) {
       InternalRow row = input.next();
       doConsume_mergeRows(row.f1, row.f2, ...);   // return lands here
       if (shouldStop()) return;                    // loop continues
     }
   
     So the return exits the wrapper function, control goes back to the while loop, and the next row is processed normally.
   
     What if the wrapping doesn't happen? (config off, or too many params) Then the code is inlined into processNext() and return would indeed exit the produce loop. But this is still correct — it's the same mechanism as if (shouldStop()) return; that every produce loop already uses. The BufferedRowIterator framework calls processNext() repeatedly, and the iterator keeps
     its position. You'd just get worse batching efficiency (one row per processNext() call instead of filling a buffer). In practice the wrapping almost always kicks in for MERGE since the operator uses all child output columns.
   
   >>> Show me some other examples of similar returns.
   
   Interesting — all those return; examples are in doProduce methods (SortExec, HashAggregateExec, SortMergeJoinExec, ShuffledHashJoinExec, Range, Columnar). Let me check if any existing operator generates a return; from doConsume, which is the pattern MergeRowsExec uses.
   
     Every existing return; in the codegen framework lives in a doProduce method, where it intentionally exits processNext(). These operators own their own iteration loops, so the return is the designed control-flow mechanism for cooperative batching:
   
     - InputRDDCodegen.doProduce (line 492) — if (shouldStop()) return; inside the while (input.hasNext()) loop
     - SortExec.doProduce (line 184) — same, inside while (sortedIterator.hasNext())
     - HashAggregateExec.doProduce (lines 561, 587, 600) — same, inside loops over the hashmap
     - SortMergeJoinExec.doProduce (lines 628, 638, 672) — same, inside nested join loops
     - ShuffledHashJoinExec.doProduce (lines 524, 542, 623, 648) — same
     - Range.doProduce (line 580) — same, with extra state-saving before return
   
     All of these are in doProduce. None of the 11 operators that override doConsume generate a return; in their consume code. MergeRowsExec would be the first.
   
     The distinction matters: a return in doProduce is inside code the operator controls — it's their own while loop in processNext(), and they handle resume state. A return in doConsume is inside code that gets injected into the child's produce loop (or into a wrapper function), and the operator generating it doesn't directly see what surrounds it. The safety relies entirely on constructDoConsumeFunction creating that wrapper boundary.
   
   >>> Without the boundary, could it theoretically exit in the middle of some produce in an "unsafe" place?
   
   Good question. Let me check what a realistic child of MergeRowsExec looks like — if it's something with a complex nested produce loop, the inlined return could be genuinely dangerous.
   
   Consider a realistic MERGE plan where MergeRowsExec sits on top of a SortMergeJoin:
   
     WholeStageCodegen
       MergeRowsExec
         SortMergeJoinExec (source JOIN target)
   
     SortMergeJoin's codegenInner at SortMergeJoinExec.scala:626 generates this structure:
   
     while (findNextJoinRows) {           // outer: advance to next key group
       // beforeLoop: set up matchIterator for this key
       while (matchIterator.hasNext()) {  // inner: iterate matched rows for this key
         InternalRow bufferedRow = matchIterator.next();
         conditionCheck
         $outputRow                       // <-- consume() lands here (line 636)
       }
       if (shouldStop()) return;          // <-- the "designed" exit point
     }
   
     The $outputRow at line 562 is ${consume(ctx, resultVars)}, which resolves to MergeRowsExec's doConsume. If that's inlined (no wrapper function), you get:
   
     while (findNextJoinRows) {
       while (matchIterator.hasNext()) {
         InternalRow bufferedRow = matchIterator.next();
         conditionCheck
         // MergeRowsExec doConsume inlined:
         if (targetPresent && sourcePresent) {
           if (keepPredicate) {
             metricUpdate;
             append(projectedRow);
             return;    // <-- exits processNext() from the INNER loop
           }
         }
       }
       if (shouldStop()) return;
     }
   
     When processNext() is called again, findNextJoinRows advances the streaming side to the next key group. The matchIterator for the previous key group is gone — any remaining matched rows for that key are silently dropped.
   
     Compare with the designed exit at line 638: if (shouldStop()) return; sits between key groups (after the inner loop completes), so all matches for a key are fully consumed before processNext() can exit.
   
     In practice this is heavily mitigated:
     - constructDoConsumeFunction is on by default and almost always applies
     - MERGE semantics require unique target keys, so each key group typically has at most one match (no remaining rows to lose)
     - SortMergeJoin's full outer join variant (codegenAntiSemiFullOuterJoin, line 843) independently wraps consume() in its own ctx.addNewFunction call, creating a boundary regardless
   
     But yes, theoretically, an inlined return from doConsume can land in the middle of a nested produce loop at a point that was never designed as a safe exit, and that could cause data loss.
   ```
   
   I think it would nevertheless be safer if you generated this in a separate function, so that the flow is fully controlled ther, and does not depend on `constructDoConsumeFunction` or this in practice landing in a place in the produce flow that is safe to return from inlined.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-53652][SQL]Codegen for MergeRowExec [spark]

Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on PR #52399:
URL: https://github.com/apache/spark/pull/52399#issuecomment-3837591801

   I think that we don't always code-gen operator and Spark has interpreted-only operators. I remember that not always code-gen-ed operators have better performance. Code-gen has its benefit but correspondingly the debugging is more difficult. The code complexity also increases.
   
   So I remember that for code-gen works we would like to see benchmark results.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org