You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "ArnavBalyan (via GitHub)" <gi...@apache.org> on 2026/08/25 14:28:53 UTC
[PR] [Spark] Introduce codegen support for whole stage merge queries [paimon]
ArnavBalyan opened a new pull request, #9390:
URL: https://github.com/apache/paimon/pull/9390
### Purpose
- Paimon processes merge queries through JVM and runs a scala iterator for every joined row.
- Add whole stage codegen for matched and unmatched actions.
### Tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Spark] Introduce codegen support for whole stage merge query [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9390:
URL: https://github.com/apache/paimon/pull/9390#issuecomment-5462521983
`constructConsumeParameters` receives all child outputs, yet Spark only guarantees that `usedInputs` have been generated. After I added an unreferenced column to the local test case, a specific Spark 3.4 test began failing consistently with the error: `Unknown variable or type "localtablescan_value_5"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Spark] Introduce codegen support for whole stage merge query [paimon]
Posted by "Akash3121 (via GitHub)" <gi...@apache.org>.
Akash3121 commented on code in PR #9390:
URL: https://github.com/apache/paimon/pull/9390#discussion_r4099196361
##########
paimon-spark/paimon-spark-ut/src/test/scala/org/apache/paimon/spark/sql/MergeRowsCodegenTestBase.scala:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.spark.sql
+
+import org.apache.paimon.spark.{PaimonSparkTestBase, SparkConnectorOptions}
+
+import org.apache.spark.sql.{PaimonUtils, Row}
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression, LessThan, Literal}
+import org.apache.spark.sql.catalyst.expressions.Literal.TrueLiteral
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows
+import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Discard, Instruction, Split}
+import org.apache.spark.sql.execution.{QueryExecution, SparkPlan, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.datasources.v2.MergeRowsExec
+import org.apache.spark.sql.paimon.Utils
+import org.apache.spark.sql.types.IntegerType
+import org.apache.spark.sql.util.QueryExecutionListener
+
+import java.util.concurrent.atomic.AtomicBoolean
+
+abstract class MergeRowsCodegenTestBase extends PaimonSparkTestBase {
+
+ import testImplicits._
+
+ private val paimonCodegenKey =
+ s"spark.paimon.${SparkConnectorOptions.MERGE_CODEGEN_ENABLED.key()}"
+
+ protected def keepInstruction(condition: Expression, output: Seq[Expression]): Instruction
+
+ test("merge row codegen requires Spark and Paimon flags") {
Review Comment:
Since `write.merge.codegen.enabled` defaults to `false`, the existing MERGE tests continue to exercise only the interpreted iterator. This new test validates one constructed Keep/Discard/Split combination, but it does not establish semantic parity for the broader MERGE surface - for example duplicate-match cardinality violations, nullable conditions and assignments, multiple ordered clauses, and `NOT MATCHED BY SOURCE` behavior.
Upstream Spark addressed the same concern by running MERGE behavior through both codegen and interpreted configurations. Could we similarly reuse or parameterize the existing Paimon MERGE tests with this option enabled, rather than relying on this single synthetic case? This would protect the new execution path against producing different rows or exceptions from the established implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Spark] Introduce codegen support for whole stage merge query [paimon]
Posted by "ArnavBalyan (via GitHub)" <gi...@apache.org>.
ArnavBalyan commented on PR #9390:
URL: https://github.com/apache/paimon/pull/9390#issuecomment-5468192368
> `constructConsumeParameters` receives all child outputs, yet Spark only guarantees that `usedInputs` have been generated. After I added an unreferenced column to the local test case, a specific Spark 3.4 test began failing consistently with the error: `Unknown variable or type "localtablescan_value_5"`.
thanks @JingsongLi! I have updated this to allow unreferenced columns and added more UT for such cases. Updated the pr thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Spark] Introduce codegen support for whole stage merge query [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #9390:
URL: https://github.com/apache/paimon/pull/9390#issuecomment-5748014991
Requirement fit: NEEDS-EVIDENCE. Implementation: FINDINGS.
The corrected `usedInputs` handling addresses the previously reproduced unreferenced-column codegen failure, so the implementation direction is plausible. However, this PR adds roughly 700 lines duplicated across Spark 3.2/3.3/3.4 execution classes and a new user-facing switch, while the only performance evidence is an upstream Spark range (1.7x–32.3x), not a Paimon MERGE workload using this implementation. Please provide a small reproducible benchmark comparing the same Paimon MERGE actions with the option on/off, including generated-code size/compile time and rows/s; that evidence is needed to justify the cross-version maintenance surface.
[P2] Regenerate the connector option documentation
The PR adds `spark.paimon.write.merge.codegen.enabled` to `SparkConnectorOptions` but does not update `docs/generated/spark_connector_configuration.html`. The Java build reaches `ConfigOptionsDocsCompletenessITCase` and fails with “Documentation is outdated”; this is a PR-caused CI failure, not infrastructure. Regenerate the Spark connector configuration and document the option's default/supported Spark versions. Other red jobs include dependency-download 429s, but this docs failure is deterministic.
I am not closing this PR because accelerating real Paimon MERGE execution can have end-to-end value; the value evidence and the failing generated-doc contract need to be completed first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org