You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "gengliangwang (via GitHub)" <gi...@apache.org> on 2026/05/17 23:29:20 UTC

[PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

gengliangwang opened a new pull request, #55934:
URL: https://github.com/apache/spark/pull/55934

   **Title**: [SPARK-56909][SQL] Refactor Cast to int/long codegen under ANSI mode
   
   **Base**: apache/spark master
   **Head**: gengliangwang:SPARK-56909-cast-int-long
   
   ---
   
   ### What changes were proposed in this pull request?
   
   Introduce `CastUtils.java` and use it from `Cast.scala` to collapse the multi-line ANSI overflow-check codegen for casts that target `int` and `long` into one-line static-method calls. Source and target `DataType` constants used in the overflow error message live as `private static final` fields on the helper class, so the happy path performs no per-row `references[]` lookups.
   
   Helpers added:
   * `longToIntExact(long)` for narrowing `long -> int`.
   * `floatToIntExact(float)`, `doubleToIntExact(double)` for fractional -> int.
   * `floatToLongExact(float)`, `doubleToLongExact(double)` for fractional -> long.
   
   `Cast.scala` changes:
   * `castIntegralTypeToIntegralTypeExactCode` and `castFractionToIntegralTypeCode` dispatch on the target type: `int` (and `long` for the fraction case) emit a `CastUtils.<...>Exact` call; byte/short targets keep the inline body (refactored in SPARK-56910).
   * Eval paths for `castToInt` add ANSI `LongType` / `FloatType` / `DoubleType` cases, and `castToLong` adds `FloatType` / `DoubleType` cases, both delegating to the new helpers.
   
   Also adds `ExpressionClassIdentitySuite` as a regression guard that `Add`, `Cast`, `EqualTo`, `And` keep their class identity after canonicalization, so a future PR cannot accidentally wrap a hot expression in `RuntimeReplaceable` and silently break optimizer pattern matches that look for those classes.
   
   ### Why are the changes needed?
   
   Part of SPARK-56908 (umbrella). The current ANSI cast codegen emits 5-line inline overflow blocks per call site. Multiplied across the many cast paths in a TPC-DS plan, this contributes meaningfully to the generated source size and to Janino compile time, and pushes whole-stage methods closer to the 64KB JVM method limit.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No. The compiled behavior is identical; only the emitted Java source text changes.
   
   ### How was this patch tested?
   
   ```
   build/sbt "catalyst/testOnly *CastSuite *CastWithAnsiOnSuite *CastWithAnsiOffSuite *AnsiCastSuite *TryCastSuite *ExpressionClassIdentitySuite"
   ```
   
   312/312 pass.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Cursor 1.x
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55934:
URL: https://github.com/apache/spark/pull/55934#issuecomment-4472860246

   
   ---
   
   ### Stack overview (SPARK-56908 umbrella)
   
   This PR is part of a stack of 8 PRs against SPARK-56908. Order:
   
   1. #55934 — [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode  *(this stack base)*
   2. #55935 — [SPARK-56910][SQL] Simplify Cast to byte/short codegen under ANSI mode
   3. #55936 — [SPARK-56911][SQL] Simplify Cast to decimal codegen under ANSI mode
   4. #55937 — [SPARK-56912][SQL] Simplify Cast to boolean codegen under ANSI mode
   5. #55939 — [SPARK-56914][SQL] Simplify decimal arithmetic codegen under ANSI mode  *(depends on #55936)*
   6. #55938 — [SPARK-56913][SQL] Simplify BinaryArithmetic byte/short codegen under ANSI mode  *(independent)*
   7. #55940 — [SPARK-56915][SQL] Simplify MakeDate/MakeInterval codegen under ANSI mode  *(independent)*
   8. #55941 — [SPARK-56916][SQL] Simplify ElementAt array codegen under ANSI mode  *(independent)*
   
   PRs 1-4 are linearly stacked on each other (each branch is based on the previous one). PR 5 (decimal arithmetic) is stacked on top of PR 3 (cast decimal) since it uses `CastUtils.changePrecisionExact`. PRs 6, 7, 8 branch off master independently.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on code in PR #55934:
URL: https://github.com/apache/spark/pull/55934#discussion_r3288515948


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:
##########
@@ -2017,23 +2025,36 @@ case class Cast(
       from: DataType,
       to: DataType): CastFunction = {
     assert(ansiEnabled)
-    val (min, max) = lowerAndUpperBound(integralType)
-    val mathClass = classOf[Math].getName
-    val fromDt = ctx.addReferenceObj("from", from, from.getClass.getName)
-    val toDt = ctx.addReferenceObj("to", to, to.getClass.getName)
-    // When casting floating values to integral types, Spark uses the method `Numeric.toInt`
-    // Or `Numeric.toLong` directly. For positive floating values, it is equivalent to `Math.floor`;
-    // for negative floating values, it is equivalent to `Math.ceil`.
-    // So, we can use the condition `Math.floor(x) <= upperBound && Math.ceil(x) >= lowerBound`
-    // to check if the floating value x is in the range of an integral type after rounding.
-    (c, evPrim, _) =>
-      code"""
-        if ($mathClass.floor($c) <= $max && $mathClass.ceil($c) >= $min) {
-          $evPrim = ($integralType) $c;
-        } else {
-          throw QueryExecutionErrors.castingCauseOverflowError($c, $fromDt, $toDt);
-        }
-      """
+    if (integralType == "int" || integralType == "long") {
+      // Float/Double -> Int/Long: call FloatExactNumeric/DoubleExactNumeric.toInt/toLong
+      // directly. Each already does the floor/ceil bounds check and throws
+      // castingCauseOverflowError -- same as the inline body.
+      val numericObj = (from match {
+        case FloatType => FloatExactNumeric
+        case DoubleType => DoubleExactNumeric
+      }).getClass.getCanonicalName.stripSuffix("$")
+      val method = s"to${integralType.capitalize}"
+      (c, evPrim, _) => code"$evPrim = $numericObj.$method($c);"
+    } else {
+      // Byte/short narrowing remains inline; refactored in a follow-up PR.

Review Comment:
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #55934:
URL: https://github.com/apache/spark/pull/55934#discussion_r3288984479


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala:
##########
@@ -1988,16 +1988,24 @@ case class Cast(
       from: DataType,
       to: DataType): CastFunction = {
     assert(ansiEnabled)
-    val fromDt = ctx.addReferenceObj("from", from, from.getClass.getName)
-    val toDt = ctx.addReferenceObj("to", to, to.getClass.getName)
-    (c, evPrim, _) =>
-      code"""
-        if ($c == ($integralType) $c) {
-          $evPrim = ($integralType) $c;
-        } else {
-          throw QueryExecutionErrors.castingCauseOverflowError($c, $fromDt, $toDt);
-        }
-      """
+    if (integralType == "int") {
+      // Long -> Int: call LongExactNumeric.toInt directly. It already does the
+      // bounds check and throws castingCauseOverflowError -- same as the inline body.
+      val numericObj = LongExactNumeric.getClass.getCanonicalName.stripSuffix("$")
+      (c, evPrim, _) => code"$evPrim = $numericObj.toInt($c);"
+    } else {
+      // Byte/short narrowing remains inline; refactored in a follow-up PR.

Review Comment:
   Nit: the symmetric `// Byte/short narrowing remains inline; refactored in a follow-up PR.` comment in `castFractionToIntegralTypeCode` was removed in your last commit (`ff9b4cb3`, "Apply suggestion from @gengliangwang") — looks like this one was missed. The else-branch's role is already clear from the `if (integralType == "int")` guard above.
   
   ```suggestion
       } else {
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55934:
URL: https://github.com/apache/spark/pull/55934#issuecomment-4520907477

   @viirya thanks for the careful review and the equivalence verification.
   
   Pushed `effb4aa` to address both readability nits:
   
   1. `castFractionToIntegralTypeCode` — the inner `from match` now has an explicit `case _ => throw SparkException.internalError(...)` default arm, so adding a new fractional type without updating this helper fails loudly at codegen time instead of silently turning into a `MatchError`.
   2. `castIntegralTypeToIntegralTypeExactCode` — the `int` branch is now structured as `from match { case LongType => LongExactNumeric; case _ => internalError }` for symmetry with `castFractionToIntegralTypeCode` and with the byte/short branches. Also added a one-line comment noting "Only LongType reaches this branch today (`castToIntCode` gates on `case LongType`)".
   
   (Plus `c966bd9` — a small fixup removing a duplicate `} else {` introduced when one of the github-ui suggestions landed.)
   
   `lint-scala` clean, 307/307 cast tests pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang commented on PR #55934:
URL: https://github.com/apache/spark/pull/55934#issuecomment-4523006760

   @cloud-fan @viirya thanks for the review.
   Merging to master/4.x


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode [spark]

Posted by "gengliangwang (via GitHub)" <gi...@apache.org>.
gengliangwang closed pull request #55934: [SPARK-56909][SQL] Simplify Cast to int/long codegen under ANSI mode
URL: https://github.com/apache/spark/pull/55934


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org