You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/04/17 00:16:09 UTC

[GitHub] [spark] bersprockets opened a new pull request, #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

bersprockets opened a new pull request, #36230:
URL: https://github.com/apache/spark/pull/36230

   ### What changes were proposed in this pull request?
   
   Change `EliminateOuterJoin#canFilterOutNull` to just return `false` instead of evaluating a `where` condition that might throw a RuntimeException.
   
   ### Why are the changes needed?
   
   Consider this query:
   ```
   select *
   from (select id, id as b from range(0, 10)) l
   left outer join (select id, id + 1 as c from range(0, 10)) r
   on l.id = r.id
   where assert_true(c > 0) is null;
   ```
   The query always fails with
   ```
   java.lang.RuntimeException: '(c#1L > cast(0 as bigint))' is not true!
   ```
   This happens even though there is no row where `c` <= 0.
   
   The `EliminateOuterJoin` rule checks if it can convert the outer join to a inner join based on the expression in the where clause, which in this case is
   ```
   assert_true(c > 0) is null
   ```
   `EliminateOuterJoin#canFilterOutNull` evaluates that expression with `c` set to `null` to see if the result is `null` or `false`. That rule doesn't expect the result to be a `RuntimeException`, but in this case it always is.
   
   That is, the assertion is failing during optimization, not at run time.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   New unit test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a diff in pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r851944618


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -144,6 +144,8 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with PredicateHelper {
     val emptyRow = new GenericInternalRow(attributes.length)
     val boundE = BindReferences.bindReference(e, attributes)
     if (boundE.exists(_.isInstanceOf[Unevaluable])) return false
+    // don't evaluate an expression that might purposefully throw an exception
+    if (boundE.exists(_.isInstanceOf[RaiseError])) return false
     val v = boundE.eval(emptyRow)

Review Comment:
   can we do a try-catch and swallow the error here? `RaiseError` doesn't match all the expressions that may fail during evaluation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bersprockets commented on pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
bersprockets commented on PR #36230:
URL: https://github.com/apache/spark/pull/36230#issuecomment-1108693619

   >do you want to open backport PRs for 3.2/3.1?
   
   Will do, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r851817750


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -144,6 +144,8 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with PredicateHelper {
     val emptyRow = new GenericInternalRow(attributes.length)
     val boundE = BindReferences.bindReference(e, attributes)
     if (boundE.exists(_.isInstanceOf[Unevaluable])) return false
+    // don't evaluate an expression that might purposefully throw an exception
+    if (boundE.exists(_.isInstanceOf[RaiseError])) return false

Review Comment:
   In fact, we should filter out all expressions that do not inherit `NoThrow` (and mark all other no-throw expressions). But that's incomplete now. Given that it already optimized all other expressions before, I am fine with this band-aid fix.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36230:
URL: https://github.com/apache/spark/pull/36230#issuecomment-1108104506

   [bersprockets](https://github.com/bersprockets) do you want to open backport PRs for 3.2/3.1?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bersprockets commented on a diff in pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r854698752


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -144,6 +144,8 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with PredicateHelper {
     val emptyRow = new GenericInternalRow(attributes.length)
     val boundE = BindReferences.bindReference(e, attributes)
     if (boundE.exists(_.isInstanceOf[Unevaluable])) return false
+    // don't evaluate an expression that might purposefully throw an exception
+    if (boundE.exists(_.isInstanceOf[RaiseError])) return false
     val v = boundE.eval(emptyRow)

Review Comment:
   >RaiseError doesn't match all the expressions that may fail during evaluation.
   
   I see what you mean. This query also fails in `EliminateOuterJoin` (because you can't have a null key):
   ```
   select *
   from (select id, id as b from range(0, 10)) l
   left outer join (select id, id + 1 as c from range(0, 10)) r
   on l.id = r.id
   where map(c, '2') is not null;
   ```
   I will add a try/catch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bersprockets commented on a diff in pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r851691251


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala:
##########
@@ -252,4 +254,19 @@ class OuterJoinEliminationSuite extends PlanTest {
       comparePlans(optimized, originalQuery.analyze)
     }
   }
+
+  // evaluating if assert_true/raised in filter predicate will cause exception
+  test("SPARK-38868: assert_true/raised in filter predicate does not throw exception") {
+    val x = testRelation.subquery(Symbol("x"))
+    val y = testRelation1.subquery(Symbol("y"))
+
+    val message = Literal(UTF8String fromString("Bad value"), StringType)
+    val originalQuery =
+      x.join(y, LeftOuter, Option("x.a".attr === "y.d".attr))
+        .where(If("y.d".attr > 0, true, RaiseError(message)).isNull)

Review Comment:
   I didn't do `AssertTrue` here because `RuntimeReplaceable` doesn't evaluate, so I just put in what `AssertTrue` gets replaced with.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] bersprockets commented on a diff in pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
bersprockets commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r853513127


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -144,6 +144,8 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with PredicateHelper {
     val emptyRow = new GenericInternalRow(attributes.length)
     val boundE = BindReferences.bindReference(e, attributes)
     if (boundE.exists(_.isInstanceOf[Unevaluable])) return false
+    // don't evaluate an expression that might purposefully throw an exception
+    if (boundE.exists(_.isInstanceOf[RaiseError])) return false
     val v = boundE.eval(emptyRow)

Review Comment:
   >can we do a try-catch and swallow the error here?
   
   Looking...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a diff in pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r855731336


##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/OuterJoinEliminationSuite.scala:
##########
@@ -252,4 +254,18 @@ class OuterJoinEliminationSuite extends PlanTest {
       comparePlans(optimized, originalQuery.analyze)
     }
   }
+
+  test("SPARK-38868: exception thrown from filter predicate does not propagate") {
+    val x = testRelation.subquery(Symbol("x"))
+    val y = testRelation1.subquery(Symbol("y"))
+
+    val message = Literal(UTF8String fromString("Bad value"), StringType)

Review Comment:
   ```suggestion
       val message = Literal(UTF8String.fromString("Bad value"), StringType)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on PR #36230:
URL: https://github.com/apache/spark/pull/36230#issuecomment-1100970799

   cc @cloud-fan FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan closed pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins
URL: https://github.com/apache/spark/pull/36230


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36230:
URL: https://github.com/apache/spark/pull/36230#issuecomment-1108104235

   thanks, merging to master/3.3!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #36230: [SPARK-38868][SQL] Avoid `RaiseError` exceptions while optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on PR #36230:
URL: https://github.com/apache/spark/pull/36230#issuecomment-1101179585

   cc @sigmod


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a diff in pull request #36230: [SPARK-38868][SQL] Don't propagate exceptions from filter predicate when optimizing outer joins

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on code in PR #36230:
URL: https://github.com/apache/spark/pull/36230#discussion_r855731223


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -144,8 +144,17 @@ object EliminateOuterJoin extends Rule[LogicalPlan] with PredicateHelper {
     val emptyRow = new GenericInternalRow(attributes.length)
     val boundE = BindReferences.bindReference(e, attributes)
     if (boundE.exists(_.isInstanceOf[Unevaluable])) return false
-    val v = boundE.eval(emptyRow)
-    v == null || v == false
+
+    // some expressions, like map(), may throw an exception when dealing with null values.
+    // therefore, we need to handle exceptions.
+    try {
+      val v = boundE.eval(emptyRow)
+      v == null || v == false
+    } catch {
+      case e: Exception =>

Review Comment:
   Let' follow the convention and catch `NonFatal(e)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org