You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "peterxcli (via GitHub)" <gi...@apache.org> on 2026/08/02 22:49:37 UTC

[PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

peterxcli opened a new pull request, #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225

   ## Which issue does this PR close?
   
   Closes #5190.
   
   ## Rationale for this change
   
   `DecimalPrecision.promote` and `CometDivide.convert` both wrapped decimal `Divide` expressions in equivalent `CheckOverflow` nodes. The native planner does not collapse that shape, so successful decimal division could perform the same precision scan twice.
   
   Promotion can also run more than once when serializers such as `ArrayContains` recursively call the public `exprToProto` API. If decimal overflow wrapping is owned by `DecimalPrecision.promote`, the transformation therefore needs to be idempotent.
   
   ## What changes are included in this PR?
   
   - Make `DecimalPrecision.promote` collapse adjacent equivalent `CheckOverflow` wrappers around decimal binary arithmetic while preserving wrappers with different result types or overflow modes.
   - Remove the specialized decimal `CheckOverflow` creation from `CometDivide.convert`, keeping overflow wrapping centralized in decimal promotion.
   - Add a real-SQL proto regression through `array_contains` covering decimal Add, Subtract, Multiply, Divide, and Remainder, including LEGACY, TRY, and ANSI divide modes.
   - Update the existing per-expression eval-mode regression to expect a single decimal Divide wrapper.
   
   ## How are these changes tested?
   
   - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu ./mvnw test -Pspark-4.1 -Dtest=none -Dsuites="org.apache.spark.sql.comet.CometDecimalArithmeticViewSuite"`
   - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu ./mvnw test -Pspark-4.2 -Dtest=none -Dsuites="org.apache.spark.sql.comet.CometDecimalArithmeticViewSuite"`
   - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu ./mvnw test -Pspark-3.4 -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite decimal_div"`
   - `JAVA_HOME=/Users/lixucheng/.sdkman/candidates/java/17.0.14-zulu ./mvnw test -Pspark-4.2 -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite decimal_div"`
   - `git diff --check`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3712097747


##########
spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala:
##########
@@ -45,6 +45,15 @@ object DecimalPrecision {
       // happen if the Spark version is < 3.4
       case e: BinaryArithmetic if e.left.prettyName == "promote_precision" => e
 
+      // Recursive exprToProto calls can re-promote every decimal binary operator below. Collapse
+      // only equivalent wrappers so the shared promotion rule remains idempotent.
+      case outer @ CheckOverflow(

Review Comment:
   filed. https://github.com/apache/datafusion-comet/issues/5248



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#issuecomment-5195126502

   @andygrove thanks for another round of review! addressed all of your review.
   
   > First, the note about approach. This PR does the opposite of what https://github.com/apache/datafusion-comet/issues/5190 proposed. It removes the serde wrapper and keeps promotion as the single owner, rather than removing the promotion wrapper. I think your choice is the better one, and it is what lets the same fix cover add, subtract, multiply, and remainder. But a reader comparing the issue to the diff will be confused without a sentence saying so.
   
   added some sentence regard the difference if #5190 (original issue) and this pr's change.
   
   > Second, the numbers. I asked for a rough before and after over a decimal-division-heavy query from tpcds-micro-benchmarks/. If you have run one, please add it. If the difference turns out to be too small to measure reliably, that is a fine answer too, but then I would rather the description lead with the idempotency and single-ownership argument and drop the framing that the duplicate scan is a meaningful cost. Either resolution works, I just do not want the perf claim sitting there unbacked.
   
   Ran benchmark with TPC-DS SF1 `store_sales` (2,879,789 rows) on an Apple M4, it shows about 6.8% improvement in time. Also added in PR description as table.
   
   > It would also help to link https://github.com/apache/datafusion-comet/issues/5248 in the description, not only in the review thread, so it stays visible from the merged commit.
   
   yes, added.
   
   > Could you add a short comment naming that dependency and linking https://github.com/apache/datafusion-comet/issues/5248, so whoever picks up that work knows this regression needs re-pointing? The promote(promote(e)) == promote(e) assertion below is independent and will still hold, so it is only the proto-shape half that is at risk.
   
   thanks for the heads-up. added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3738575791


##########
spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala:
##########
@@ -45,6 +45,15 @@ object DecimalPrecision {
       // happen if the Spark version is < 3.4
       case e: BinaryArithmetic if e.left.prettyName == "promote_precision" => e

Review Comment:
   Thanks, addressed in https://github.com/apache/datafusion-comet/pull/5225/commits/d6625e134127b8606e7e9d9073da3edc565280af
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3738222125


##########
spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala:
##########
@@ -45,6 +45,15 @@ object DecimalPrecision {
       // happen if the Spark version is < 3.4
       case e: BinaryArithmetic if e.left.prettyName == "promote_precision" => e

Review Comment:
   Now that the serde no longer adds its own wrapper, `promote` is the only thing standing between a decimal `Divide` and the raw `i128::MAX` sentinel. That makes this branch worth a second look, because it is the one arm that returns early without wrapping.
   
   The comment says it can only fire on Spark versions below 3.4, and `QueryPlanSerde.scala:911` says the same thing about 3.3. `PromotePrecision` was removed by SPARK-39316 in 3.4.0, and `pom.xml` puts the floor at 3.4.3, so this cannot match on any version Comet builds against. It is also the one remaining shape where a decimal `Divide` reaches `CometDivide.convert` with both operands decimal and no `CheckOverflow` anywhere, which is exactly the silent-garbage case the new `getSupportLevel` guard was added to rule out. That guard cannot catch it, since the operands are decimal there.
   
   Could this branch just be deleted? It is three lines, and removing it makes the invariant total rather than nearly total. If you would rather not touch an unrelated arm of the rule in this PR, a tracking issue linked here works too. I would just not leave it as-is, now that it is load-bearing in a way it was not before.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3712225519


##########
spark/src/main/scala/org/apache/comet/serde/arithmetic.scala:
##########
@@ -263,7 +263,7 @@ object CometDivide extends CometExpressionSerde[Divide] with MathBase {
     // For now, use NullIf to swap zeros with nulls.
     val rightExpr =
       if (expr.evalMode != EvalMode.ANSI) nullIfWhenPrimitive(expr.right) else expr.right
-    val divideExpr = createMathExpression(
+    createMathExpression(

Review Comment:
   added a sanity check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3706832148


##########
spark/src/main/scala/org/apache/comet/serde/arithmetic.scala:
##########
@@ -263,7 +263,7 @@ object CometDivide extends CometExpressionSerde[Divide] with MathBase {
     // For now, use NullIf to swap zeros with nulls.
     val rightExpr =
       if (expr.evalMode != EvalMode.ANSI) nullIfWhenPrimitive(expr.right) else expr.right
-    val divideExpr = createMathExpression(
+    createMathExpression(

Review Comment:
   One thing I want to check here. The guard you removed fired on `expr.dataType.isInstanceOf[DecimalType]` alone. The `promote` rule requires both operands to match `DecimalExpression` as well, and `getSupportLevel` only looks at `expr.left.dataType`. So if a decimal-typed `Divide` with a non-decimal operand ever reaches this serde, there is no `CheckOverflow` anywhere and the native `i128::MAX` sentinel comes back as a real value.
   
   I could not construct that shape through Spark's type coercion, so I believe it is unreachable today. Since the failure mode is silent wrong data rather than a fallback, would you be up for a defensive guard? Returning `Unsupported` from `getSupportLevel` when `expr.dataType` is decimal but the operands would not match `promote`'s pattern would keep the invariant checkable next to the code that depends on it.



##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometDecimalArithmeticViewSuite.scala:
##########
@@ -119,4 +104,38 @@ class CometDecimalArithmeticViewSuite extends CometTestBase {
       }
     }
   }
+
+  test("issue #5190: recursive serialization does not duplicate decimal CheckOverflow") {
+    val left = "CAST(id AS DECIMAL(10, 0))"
+    val right = "CAST(id + 1 AS DECIMAL(10, 0))"
+    val operations: Seq[(String, Boolean, String, ExprOuterClass.Expr => Boolean, Boolean)] = Seq(

Review Comment:
   This tuple has two unlabeled booleans in it, and `("divide TRY", true, ..., false)` is hard to read without counting positions back to the destructuring on the next block. A small case class with named fields, or moving `ansiEnabled` and `failOnError` next to their names, would make this easier to extend when someone adds a case later.



##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometDecimalArithmeticViewSuite.scala:
##########
@@ -119,4 +104,38 @@ class CometDecimalArithmeticViewSuite extends CometTestBase {
       }
     }
   }
+
+  test("issue #5190: recursive serialization does not duplicate decimal CheckOverflow") {

Review Comment:
   This regression does not depend on anything 4.1-specific, but sitting under `spark-4.1+` means 3.4, 3.5, and 4.0 CI never runs it, and the duplication it guards against exists on all of them. Could it move to a version-agnostic suite under `spark/src/test/scala/org/apache/spark/sql/comet/`? A new `CometDecimalPromotionSuite` would work. The other two tests in this file can stay where they are since they need `NumericEvalContext`.
   
   While it is moving, two additions would be worth it. ANSI is only exercised for `divide`, but the collapse guard compares `nullOnOverflow`, so ANSI is exactly the case that would catch a wrong guard for add, subtract, multiply, and remainder. And a direct assertion that `promote(promote(e)) == promote(e)` would state the idempotency property outright instead of leaving it to be inferred from the `array_contains` proto shape.



##########
spark/src/main/scala/org/apache/spark/sql/comet/DecimalPrecision.scala:
##########
@@ -45,6 +45,15 @@ object DecimalPrecision {
       // happen if the Spark version is < 3.4
       case e: BinaryArithmetic if e.left.prettyName == "promote_precision" => e
 
+      // Recursive exprToProto calls can re-promote every decimal binary operator below. Collapse
+      // only equivalent wrappers so the shared promotion rule remains idempotent.
+      case outer @ CheckOverflow(

Review Comment:
   This is a good fix, and the bottom-up collapse converges correctly at any depth.
   
   The re-promotion it works around comes from serdes calling the public `exprToProto` on children that `promote` has already visited. There are 64 such call sites in the serde package. The aggregate ones genuinely need the promoting entry point, but the ones in `arrays.scala` and `bitwise.scala` are re-walking an already-promoted tree, and each walk drags `liftCoverageTags` along with it. That is separate work from this PR, but could you file a tracking issue for it and link it here? Otherwise it will not get picked up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3721821918


##########
spark/src/test/scala/org/apache/spark/sql/comet/CometDecimalPromotionSuite.scala:
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.spark.sql.comet
+
+import org.apache.spark.sql.CometTestBase
+import org.apache.spark.sql.catalyst.expressions.{ArrayContains, AttributeReference, Divide, EvalMode}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{DecimalType, IntegerType}
+
+import org.apache.comet.serde.{CometDivide, ExprOuterClass, QueryPlanSerde, Unsupported}
+
+class CometDecimalPromotionSuite extends CometTestBase {
+
+  private case class Operation(
+      name: String,
+      symbol: String,
+      hasProto: ExprOuterClass.Expr => Boolean)
+
+  private case class TestMode(name: String, ansiEnabled: Boolean, failOnError: Boolean)
+
+  test("issue #5190: decimal promotion is idempotent during recursive serialization") {
+    val left = "CAST(id AS DECIMAL(10, 0))"
+    val right = "CAST(id + 1 AS DECIMAL(10, 0))"
+    val divide = Operation(name = "divide", symbol = "/", hasProto = _.hasDivide)
+    val operations = Seq(
+      Operation(name = "add", symbol = "+", hasProto = _.hasAdd),
+      Operation(name = "subtract", symbol = "-", hasProto = _.hasSubtract),
+      Operation(name = "multiply", symbol = "*", hasProto = _.hasMultiply),
+      divide,
+      Operation(name = "remainder", symbol = "%", hasProto = _.hasRemainder))
+    val legacy = TestMode(name = "LEGACY", ansiEnabled = false, failOnError = false)
+    val ansi = TestMode(name = "ANSI", ansiEnabled = true, failOnError = true)
+
+    def check(operation: Operation, mode: TestMode, arithmetic: String): Unit = {
+      val name = s"${operation.name} ${mode.name}"
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> mode.ansiEnabled.toString) {
+        val plan = spark
+          .sql(s"SELECT array_contains(array($arithmetic), $arithmetic) FROM range(1, 4)")

Review Comment:
   One thing worth guarding against here. The re-promotion this test exercises only happens because `CometArrayContains` serializes its children through the public `exprToProto`. That is exactly what #5248 proposes to change. Once someone switches `arrays.scala` to `exprToProtoInternal`, the `array_contains` half of this test stops covering the duplicate-wrapper path and keeps passing silently.
   
   Could you add a short comment naming that dependency and linking #5248, so whoever picks up that work knows this regression needs re-pointing? The `promote(promote(e)) == promote(e)` assertion below is independent and will still hold, so it is only the proto-shape half that is at risk.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: avoid duplicate CheckOverflow evaluation for decimal division [datafusion-comet]

Posted by "peterxcli (via GitHub)" <gi...@apache.org>.
peterxcli commented on code in PR #5225:
URL: https://github.com/apache/datafusion-comet/pull/5225#discussion_r3712222941


##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometDecimalArithmeticViewSuite.scala:
##########
@@ -119,4 +104,38 @@ class CometDecimalArithmeticViewSuite extends CometTestBase {
       }
     }
   }
+
+  test("issue #5190: recursive serialization does not duplicate decimal CheckOverflow") {

Review Comment:
   - moved to `CometDecimalPromotionSuite`. 
   - make ANSI also exercise for `add`, `subtract`, `multiply`, and `remainder`
   - addded `promote(promote(e)) == promote(e)`



##########
spark/src/test/spark-4.1+/org/apache/spark/sql/comet/CometDecimalArithmeticViewSuite.scala:
##########
@@ -119,4 +104,38 @@ class CometDecimalArithmeticViewSuite extends CometTestBase {
       }
     }
   }
+
+  test("issue #5190: recursive serialization does not duplicate decimal CheckOverflow") {
+    val left = "CAST(id AS DECIMAL(10, 0))"
+    val right = "CAST(id + 1 AS DECIMAL(10, 0))"
+    val operations: Seq[(String, Boolean, String, ExprOuterClass.Expr => Boolean, Boolean)] = Seq(

Review Comment:
   refactored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org