(spark) branch branch-4.2 updated: [SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable

[ho...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

asf-gitbox-commits pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new fac17851a955 [SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable
fac17851a955 is described below

commit fac17851a95590d304df194a2c160555f4be2cc8
Author: Holden Karau <ho...@pigscanfly.ca>
AuthorDate: Fri Aug 21 12:32:27 2026 -0700

    [SPARK-58627][SQL] Mark raise error Throwable & fix sequence throwable
    
    Corrects the `Expression.throwable` metadata on two expressions:
    
    1. Overrides `throwable` to `true` on `RaiseError`.
    2. Makes `Sequence`'s existing override fall back to its children rather than discarding the inherited default.
    
    `Expression.throwable` (added in SPARK-46707) is opt-in metadata that tells the optimizer an expression may raise a runtime error, so a predicate containing it must not be relocated to a position where it runs on rows the original plan would not have evaluated it on. `RaiseError` always throws when evaluated but never declared the flag, so it inherited `children.exists(_.throwable)` -- false for the usual case of a literal error class and parameters.
    
    `Sequence` was the only expression in the tree overriding the flag, and it did so as `stepOpt.isDefined`, dropping the inherited `children.exists(_.throwable)` term entirely. A throwing child under a stepless `sequence(...)` therefore reported non-throwable, which would also have masked the new flag on `RaiseError`.
    
    Without the flag, `CombineFilters` and `PushPredicateThroughJoin` treat a predicate containing `raise_error` (or `assert_true`, which is rewritten to `If(cond, null, RaiseError(...))`) as freely movable. Pushing such a predicate below a selective join, or merging it into a filter that would have removed the offending rows, can make a query fail at runtime that previously succeeded.
    
    The join below matches no rows, so the predicate should never be evaluated. But the predicate has no column references, so `references.subsetOf(left.outputSet)` holds trivially and it is pushed onto the left side, where it fires on the first row of `t1`:
    
    ```sql
    CREATE OR REPLACE TEMP VIEW t1 AS SELECT * FROM VALUES (1), (2), (3) AS t(a);
    CREATE OR REPLACE TEMP VIEW t2 AS SELECT * FROM VALUES (4), (5), (6) AS t(b);
    SELECT * FROM t1 JOIN t2 ON t1.a = t2.b WHERE raise_error('boom') IS NULL;
    -- [USER_RAISED_EXCEPTION] boom SQLSTATE: P0001
    ```
    
    With this change the query returns an empty result.
    
    Yes, as a bug fix. A predicate containing `raise_error` or `assert_true` is no longer pushed through a join, pushed into a join condition, or combined with an adjacent filter, so the error is raised only on the rows the unoptimized plan would have evaluated it on. Queries that previously failed spuriously now succeed. The same now holds for a throwing expression nested under a stepless `sequence(...)`.
    
    Added UTs:
    - `MiscExpressionsSuite`: asserts the flag on `RaiseError`
    - `FilterPushdownSuite`: a `raise_error` predicate is not pushed through a join and not combined with an adjacent filter, each paired with a non-throwing predicate of the same shape that still is.
    
    Generated-by: Claude Code (Opus 5) human review and cleanup after by Holden
    
    Closes #57975 from holdenk/SPARK-58627-mark-raise-error-throwable.
    
    Lead-authored-by: Holden Karau <ho...@pigscanfly.ca>
    Co-authored-by: Holden Karau <ho...@snowflake.com>
    Signed-off-by: Holden Karau <ho...@snowflake.com>
    (cherry picked from commit 7f10fc4107168747f4b0599fc35e2e4e7a317519)
    Signed-off-by: Holden Karau <ho...@snowflake.com>
---
 .../apache/spark/sql/catalyst/expressions/collectionOperations.scala  | 4 ++--
 .../main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala   | 1 +
 .../apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala  | 4 ++++
 3 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index a275d7bb1055..ee1f61ecd4f3 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -3297,8 +3297,8 @@ case class Sequence(
 
   override def nullable: Boolean = children.exists(_.nullable)
 
-  // If step is defined, then an error will be thrown if the start and stop do not satisfy the step.
-  override lazy val throwable: Boolean = stepOpt.isDefined
+  // Can throw if step is defined and start and stop don't match or any of the children can throw.
+  override lazy val throwable: Boolean = stepOpt.isDefined || children.exists(_.throwable)
 
   override def dataType: ArrayType = ArrayType(start.dataType, containsNull = false)
 
diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
index 23e8b343e431..d237f212d91f 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/misc.scala
@@ -84,6 +84,7 @@ case class RaiseError(errorClass: Expression, errorParms: Expression, dataType:
 
   override def foldable: Boolean = false
   override def nullable: Boolean = true
+  override lazy val throwable: Boolean = true
   override def inputTypes: Seq[AbstractDataType] =
     Seq(
       StringTypeWithCollation(supportsTrimCollation = true),
diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
index ff0183eb265b..e79221042118 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/MiscExpressionsSuite.scala
@@ -55,6 +55,10 @@ class MiscExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper {
     )
   }
 
+  test("SPARK-58627: RaiseError is throwable") {
+    assert(RaiseError(Literal("error!")).throwable)
+  }
+
   test("SPARK-55109: RaiseError.sql uses single-argument form only for known error classes") {
     assert(RaiseError(Literal("error!")).sql === "raise_error('error!')")
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org