You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/28 02:13:03 UTC
[GitHub] [spark] sunchao opened a new pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
sunchao opened a new pull request #29567:
URL: https://github.com/apache/spark/pull/29567
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
The following if clause:
```
if(p, null, false)
```
can be simplified to:
```
and(p, null)
```
Similarly, the clause:
```
if(p, null, true)
```
can be simplified to
```
or(p, null)
```
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Converting if to or/and clauses can better push filters down.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Unit tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684037016
Merged into master. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684031237
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r480325897
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,30 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ checkEvaluation(If(b, nullLiteral, FalseLiteral), And(b, nullLiteral).eval(EmptyRow))
+ checkEvaluation(If(b, nullLiteral, TrueLiteral), Or(Not(b), nullLiteral).eval(EmptyRow))
+ }
+
+ // should have no effect on expressions with nullable if condition
+ assert((Factorial(5) > 100L).nullable)
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ checkEvaluation(If(Factorial(5) > 100L, nullLiteral, b),
+ If(Factorial(5) > 100L, nullLiteral, b).eval(EmptyRow))
+ }
+ }
Review comment:
the indentation is off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682991489
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127998/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683219924
cc @dbtsai @viirya
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682365574
**[Test build #127976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127976/testReport)** for PR 29567 at commit [`f9e87f0`](https://github.com/apache/spark/commit/f9e87f07d7bc8020d0f1409019dafe01487f41ad).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479792044
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
##########
@@ -221,14 +221,14 @@ class BooleanSimplificationSuite extends PlanTest with ExpressionEvalHelper with
test("Complementation Laws - null handling") {
checkCondition('e && !'e,
- testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), false)).analyze)
+ testRelationWithData.where(And(Literal(null, BooleanType), 'e.isNull)).analyze)
Review comment:
Usually we only test corresponding rule in each test suite. But some suites use multiple rules like this one. For this one, although we can still only test `BooleanSimplification` if we add another optimizer, but seems too much. Maybe currently it's good enough. Maybe add a comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479714579
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Thanks for being patient with me. I think this new pattern looks okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682286525
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682461280
I think it should be correct now, but it would be nice to have some checks in the tests.
Are there equivalent cases for `if(p, false, null)` and `if(p, true, null)`?
Perhaps instead of a new `Rule` the cases could be added to the existing `SimplifyConditionals`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684133047
**[Test build #128118 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128118/testReport)** for PR 29567 at commit [`71bd033`](https://github.com/apache/spark/commit/71bd033bfa58365ceabd108a791fd0748a27e1e0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479869056
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
Review comment:
In eval, can we check when b is nullable too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683147685
**[Test build #128002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128002/testReport)** for PR 29567 at commit [`cc66198`](https://github.com/apache/spark/commit/cc661984f3ccbf59bbabb91c7d92b17524ab74d3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479869056
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
Review comment:
minor, in eval, can we check when b is nullable too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683214869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479697903
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Hmm, I'm not sure why `Or(Not(p), null)` can't be pushed down. I have this test case:
```scala
test("test pushdown if") {
withTempPath( dir => {
spark.range(1, 100).toDF("c").write.parquet(dir.getAbsolutePath)
withTempView("t") {
spark.read.parquet(dir.getAbsolutePath).createOrReplaceTempView("t")
sql("SELECT * FROM t WHERE if(isnull(c), null, true)").explain()
}
})
}
```
Without the rule, the explain result is:
```
*(1) Filter if (isnull(id#223L)) false else true
+- *(1) ColumnarToRow
+- FileScan parquet [id#223L] Batched: true, DataFilters: [if (isnull(id#223L)) false else true], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/z3/ptkgr4kn4pv9v8g7s1fnr5mm0000gn/T/spark-858a579d-3a..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:bigint>
```
with the rule it become:
```
== Physical Plan ==
*(1) Filter isnotnull(id#223L)
+- *(1) ColumnarToRow
+- FileScan parquet [id#223L] Batched: true, DataFilters: [isnotnull(id#223L)], Format: Parquet, Location: InMemoryFileIndex[file:/private/var/folders/z3/ptkgr4kn4pv9v8g7s1fnr5mm0000gn/T/spark-08e9254e-df..., PartitionFilters: [], PushedFilters: [IsNotNull(id)], ReadSchema: struct<id:bigint>
```
so you can see the `Not(p)` here is pushed down to data sources. Did I miss anything?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684030359
**[Test build #128118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128118/testReport)** for PR 29567 at commit [`71bd033`](https://github.com/apache/spark/commit/71bd033bfa58365ceabd108a791fd0748a27e1e0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479937684
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
Review comment:
yes will add a test case for this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683390557
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682997271
**[Test build #127999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127999/testReport)** for PR 29567 at commit [`bb7c38c`](https://github.com/apache/spark/commit/bb7c38cdd5f139a7cbcf04de6ed59d8fd0313bd4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682366097
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127976/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479833572
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
For a Spark `Expression`, deterministic means that for fixed input it should always return the same result (e.g., a deterministic function), while `nullable` means whether it will be evaluated to null. In this case we need the latter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683628938
**[Test build #128086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128086/testReport)** for PR 29567 at commit [`2b90c19`](https://github.com/apache/spark/commit/2b90c195a42300e981589ba5c946d95156c24342).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684030359
**[Test build #128118 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128118/testReport)** for PR 29567 at commit [`71bd033`](https://github.com/apache/spark/commit/71bd033bfa58365ceabd108a791fd0748a27e1e0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682395070
Thanks for checking. I made a few mistakes in the draft PR. Firstly, the second rule should be:
```
if(p, null, true) => or(not(p), null)
```
Secondly, the predicate `p` should be deterministic and evaluate to either true or false, but not null. I'll see how we can put restriction that.
Since three-valued logic has the following properties:
| a | b | a and b | a or b |
| --| --| ----| ----|
| true | null | null | true |
| false | null | false | null |
I think the above should be semantically equivalent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai edited a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai edited a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683259420
This will be a very useful optimization to turn `if causes` into expressions that can be pushdown. For `CaseWhen` if there is only one branch, we can convert it into `If`. A PR awhile ago https://github.com/apache/spark/pull/21850 implemented this optimization but didn't end up merging into master because at that time, `CaseWhen` and `If` have not much performance difference. Maybe we can revisit that again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683444220
Looks okay. Leave to @dongjoon-hyun for another check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682990514
**[Test build #127998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127998/testReport)** for PR 29567 at commit [`e4d929c`](https://github.com/apache/spark/commit/e4d929cae26b0e180ba31830fa32949e1dc44b60).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai closed pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai closed pull request #29567:
URL: https://github.com/apache/spark/pull/29567
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683159589
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479868516
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
Review comment:
Can we check when cond is nullable, we should not do the optimization?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682286525
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682375454
Are you sure these rules are correct?
The first one seems to fail if p is false
And the seconds one seems to fail on all 3 values
I tried a quick test and 4 out of 6 of these failed.
```
val nullLiteral = Literal(null, BooleanType)
Seq(Literal.TrueLiteral, Literal.FalseLiteral, nullLiteral).foreach(p => {
assertEquivalent(If(p, nullLiteral, Literal.FalseLiteral), And(p, nullLiteral))
assertEquivalent(If(p, nullLiteral, Literal.TrueLiteral), Or(p, nullLiteral))
})
private def assertEquivalent(e1: Expression, e2: Expression): Unit = {
test("SPARK-32721: " + e1.toString) {
checkEvaluation (e1, e2.eval (EmptyRow) )
}
}
```
It seem that there might be some sort of equivalence here, but I'm not 100% sure how it should look like.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479687170
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Pushdown predicate is somehow different to general predicate. For example, IIUC we won't push down `Or(Not(p), null)` because of the null literal. Predicate pushdown only rewrites predicates applied to a field column, e.g. col > 1.
Just for predicate pushdown, maybe we can transfer `Or(Not(p), null)` to just `Not(p)`? Because if `p` is true, the predicate evaluates to null, then we filter out the row. If `p` is false, the predicate value is true. So it semantically equals to `Not(p)` for predicate used in pushdown.
I'm not sure if I miss anything or edge case, and this also doesn't work for nested predicate. Besides, this is unlikely case too I think, we can consider if it's worth the changing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684133830
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683417566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683000805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479665954
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Hm, I think for expression rewriting, this won't be much faster. This is like @HyukjinKwon said, should be a rare case. The most likely usage, I think, is to turn a predicate to a form being able to push down to data source.
Adding a rule is too much. As part of existing rule looks more proper.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683778917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683629587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683159589
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683525201
> cc @gatorsmile since he expressed his concerns at that time.
I think the concern @gatorsmile had was in https://github.com/apache/spark/pull/21850, converting `CaseWhen` to `If`, we will not get too much performance gain. However, this PR is trying to convert `If` to `And` or `Or` when possible, so we are able to pushdown more into data source. With this merged, we could revisit https://github.com/apache/spark/pull/21850 since it opens up the possibility that when we have one branch in `CaseWhen`, we can potentially optimize it to `And` or `Or`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479821699
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
`cond.deterministic` seems to be required, doesn't it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479734454
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { p =>
+ checkEvaluation(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral).eval(EmptyRow))
Review comment:
2 spaces indentation
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683148169
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683259420
This will be a very useful optimization to turn if causes into expressions that can be pushdown. For `CaseWhen` if there is only one branch, we can convert it into `If`. A PR before https://github.com/apache/spark/pull/21850 tried this optimization but didn't end up merging into master because at that time, `CaseWhen` and `If` have not much performance difference. Maybe we can revisit it again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479736781
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
##########
@@ -221,14 +221,14 @@ class BooleanSimplificationSuite extends PlanTest with ExpressionEvalHelper with
test("Complementation Laws - null handling") {
checkCondition('e && !'e,
- testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), false)).analyze)
+ testRelationWithData.where(And(Literal(null, BooleanType), 'e.isNull)).analyze)
Review comment:
Yeah. What is the recommended way for adding test suite for optimizers? should each test suite only use the corresponding optimizer rule, or it's OK to also add those relevant ones?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683390557
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683148169
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683214525
**[Test build #128002 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128002/testReport)** for PR 29567 at commit [`cc66198`](https://github.com/apache/spark/commit/cc661984f3ccbf59bbabb91c7d92b17524ab74d3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683391066
**[Test build #128040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128040/testReport)** for PR 29567 at commit [`60e0e92`](https://github.com/apache/spark/commit/60e0e925957ff3e1b6e240792ad798e748662173).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479867566
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
FYI,
```
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
```
This rule is an example why `cond` has to be deterministic in other case that Dongjoon raised the concern.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479935617
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
I see. Thanks @dbtsai @dongjoon-hyun and @viirya , yeah I think in this case we don't require `cond` to be deterministic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479863466
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
@sunchao I think the concern @dongjoon-hyun has is it's possible that `cond` is non-deterministic or has side-effect (all the expressions that have side effects in Spark, we consider them as non-deterministic). If we naively simplify them, we might change the final result.
For example, `If(cond, TrueLiteral, TrueLiteral)` can be always simplified to `TrueLiteral`; however, this will skip evaluating `cond` which might have side-effect such as counting the number of times we call `cond`.
In the first case, the `cond` in `And(cond, l)` will be alway evaluated, and in the second case, `Or(Not(cond), l)`, since `l` will never be `true`, `Not(cond)` will always be evaluated.
As a result, maybe we don't need such strong requirement that `cond` has to be deterministic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683159601
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127999/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682991289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479869153
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
In the added patterns, `cond` isn't skipped or changed with evaluation order, I think `cond.deterministic` shouldn't be required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479732081
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala
##########
@@ -221,14 +221,14 @@ class BooleanSimplificationSuite extends PlanTest with ExpressionEvalHelper with
test("Complementation Laws - null handling") {
checkCondition('e && !'e,
- testRelationWithData.where(If('e.isNull, Literal.create(null, BooleanType), false)).analyze)
+ testRelationWithData.where(And(Literal(null, BooleanType), 'e.isNull)).analyze)
Review comment:
Although I think it is better to test `BooleanSimplification` like original, but `BooleanSimplificationSuite`'s optimizer mixes `SimplifyConditionals` and `BooleanSimplification`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684031237
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-684133830
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479733988
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
We typically use `cond` in the `If` expression. Maybe we can do the following.
```
case If(cond, Literal(null, _), FalseLiteral) if !p.nullable => And(cond, Literal(null, _))
case If(cond, Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(cond), Literal(null, _))
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683417338
**[Test build #128040 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128040/testReport)** for PR 29567 at commit [`60e0e92`](https://github.com/apache/spark/commit/60e0e925957ff3e1b6e240792ad798e748662173).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683387481
LGTM except couple minor comments. +@dongjoon-hyun for another round of review before we merge it. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682991289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r480654412
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
Review comment:
ah it's covered in the followup https://github.com/apache/spark/pull/29603
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683000805
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682286294
**[Test build #127976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127976/testReport)** for PR 29567 at commit [`f9e87f0`](https://github.com/apache/spark/commit/f9e87f07d7bc8020d0f1409019dafe01487f41ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682991480
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao edited a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao edited a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682395070
Thanks for checking. I made a few mistakes in the draft PR. Firstly, the second rule should be:
```
if(p, null, true) => or(not(p), null)
```
Secondly, the predicate `p` should be deterministic and evaluate to either true or false, but not null. I'll see how we can put restriction that, and add more tests.
Since three-valued logic has the following properties:
| a | b | a and b | a or b |
| --| --| ----| ----|
| true | null | null | true |
| false | null | false | null |
I think the above should be semantically equivalent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682286294
**[Test build #127976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127976/testReport)** for PR 29567 at commit [`f9e87f0`](https://github.com/apache/spark/commit/f9e87f07d7bc8020d0f1409019dafe01487f41ad).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682990514
**[Test build #127998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127998/testReport)** for PR 29567 at commit [`e4d929c`](https://github.com/apache/spark/commit/e4d929cae26b0e180ba31830fa32949e1dc44b60).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683777378
**[Test build #128086 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128086/testReport)** for PR 29567 at commit [`2b90c19`](https://github.com/apache/spark/commit/2b90c195a42300e981589ba5c946d95156c24342).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682366092
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479668552
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Hmm, I'm rethinking this. Is a predicate `And(p, null)` be pushed down? Predicate pushdown won't rewrite null literals.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479734534
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { p =>
Review comment:
should we use the same name in the foreach?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683159297
**[Test build #127999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127999/testReport)** for PR 29567 at commit [`bb7c38c`](https://github.com/apache/spark/commit/bb7c38cdd5f139a7cbcf04de6ed59d8fd0313bd4).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683629587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682377257
I don't look into it yet, but the motivation is the same with https://github.com/apache/spark/pull/27231 ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683778917
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683147685
**[Test build #128002 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128002/testReport)** for PR 29567 at commit [`cc66198`](https://github.com/apache/spark/commit/cc661984f3ccbf59bbabb91c7d92b17524ab74d3).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682991459
**[Test build #127998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127998/testReport)** for PR 29567 at commit [`e4d929c`](https://github.com/apache/spark/commit/e4d929cae26b0e180ba31830fa32949e1dc44b60).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479209554
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -716,6 +716,17 @@ object SimplifyCasts extends Rule[LogicalPlan] {
}
}
+/**
+ * Simplify if clauses of pattern `if(p, null, true|false)`, by replacing them with
+ * AND or OR clauses which are simpler and can better be pushed down
+ */
+object SimplifyIf extends Rule[LogicalPlan] {
+ val nullLiteral = Literal(null, BooleanType)
+ def apply(plan: LogicalPlan): LogicalPlan = plan transformAllExpressions {
+ case If(p, Literal(null, _), FalseLiteral) => And(p, nullLiteral)
+ case If(p, Literal(null, _), TrueLiteral) => Or(p, nullLiteral)
Review comment:
Is it common enough to write an if with a `null` with returning a bool? Adding a rule isn't free and it gives some overhead to all the queries basically during the planning. I would avoid doing it if it's pretty unlikely.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683446609
Thank you, @sunchao , @dbtsai , @viirya .
cc @gatorsmile since he expressed his concerns at that time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479867566
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
+ case If(cond, l @ Literal(null, _), TrueLiteral) if !cond.nullable => Or(Not(cond), l)
Review comment:
FYI,
```
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
```
This rule is an example why `cond` has to be deterministic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479674614
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
I'm yet to find a case where `If(p, null, false)` gets populated to datasource. I think most of the time they just get replaced by `false` so transforming them to `And(p, null)` doesn't seem to help much.
On the other hand, `Or(Not(p), null)` does get pushed down to datasources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479700612
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Oh, I see.
```
== Analyzed Logical Plan ==
c: bigint
Project [c#5849L]
+- Filter if (isnull(c#5849L)) cast(null as boolean) else true
+- SubqueryAlias t
+- Relation[c#5849L] parquet
== Optimized Logical Plan ==
Filter isnotnull(c#5849L)
+- Relation[c#5849L] parquet
```
The predicate actually will be optimized to a simple `IsNotNull`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479701742
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
> Just for predicate pushdown, maybe we can transfer Or(Not(p), null) to just Not(p)?
Then `Or(Not(p), null)` is already optimized to `Not(p)` now for predicate pushdown. But this is only for non-nested predicate case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682395070
Thanks for checking. I made a few mistakes in the PR. Firstly, the second rule should be:
```
if(p, null, true) => or(not(p), null)
```
Secondly, the predicate `p` should be deterministic and evaluate to either true or false, but not null. I'll see how we can put restriction that.
Since three-valued logic has the following properties:
| a | b | a and b | a or b |
| --| --| ----| ----|
| true | null | null | true |
| false | null | false | null |
I think the above should be semantically equivalent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479737649
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyConditionalSuite.scala
##########
@@ -165,4 +165,23 @@ class SimplifyConditionalSuite extends PlanTest with PredicateHelper {
Literal(1))
)
}
+
+ test("simplify if when then clause is null and else clause is boolean") {
+ val p = IsNull('a)
+ val nullLiteral = Literal(null, BooleanType)
+ assertEquivalent(If(p, nullLiteral, FalseLiteral), And(p, nullLiteral))
+ assertEquivalent(If(p, nullLiteral, TrueLiteral), Or(IsNotNull('a), nullLiteral))
+
+ // the rule should not apply to nullable predicate
+ Seq(TrueLiteral, FalseLiteral).foreach { b =>
+ assertEquivalent(If(GreaterThan('a, 42), nullLiteral, b),
+ If(GreaterThan('a, 42), nullLiteral, b))
+ }
+
+ // check evaluation also
+ Seq(TrueLiteral, FalseLiteral).foreach { p =>
Review comment:
Sure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683391066
**[Test build #128040 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128040/testReport)** for PR 29567 at commit [`60e0e92`](https://github.com/apache/spark/commit/60e0e925957ff3e1b6e240792ad798e748662173).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682997271
**[Test build #127999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127999/testReport)** for PR 29567 at commit [`bb7c38c`](https://github.com/apache/spark/commit/bb7c38cdd5f139a7cbcf04de6ed59d8fd0313bd4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683214869
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk edited a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
tanelk edited a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682375454
Are you sure these rules are correct?
The first one seems to fail if p is null
And the seconds one seems to fail on all 3 values
I tried a quick test and 4 out of 6 of these failed.
```
val nullLiteral = Literal(null, BooleanType)
Seq(Literal.TrueLiteral, Literal.FalseLiteral, nullLiteral).foreach(p => {
assertEquivalent(If(p, nullLiteral, Literal.FalseLiteral), And(p, nullLiteral))
assertEquivalent(If(p, nullLiteral, Literal.TrueLiteral), Or(p, nullLiteral))
})
private def assertEquivalent(e1: Expression, e2: Expression): Unit = {
test("SPARK-32721: " + e1.toString) {
checkEvaluation (e1, e2.eval (EmptyRow) )
}
}
```
It seem that there might be some sort of equivalence here, but I'm not 100% sure how it should look like.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683628938
**[Test build #128086 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128086/testReport)** for PR 29567 at commit [`2b90c19`](https://github.com/apache/spark/commit/2b90c195a42300e981589ba5c946d95156c24342).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-683417566
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479352330
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -716,6 +716,17 @@ object SimplifyCasts extends Rule[LogicalPlan] {
}
}
+/**
+ * Simplify if clauses of pattern `if(p, null, true|false)`, by replacing them with
+ * AND or OR clauses which are simpler and can better be pushed down
+ */
+object SimplifyIf extends Rule[LogicalPlan] {
+ val nullLiteral = Literal(null, BooleanType)
+ def apply(plan: LogicalPlan): LogicalPlan = plan transformAllExpressions {
+ case If(p, Literal(null, _), FalseLiteral) => And(p, nullLiteral)
+ case If(p, Literal(null, _), TrueLiteral) => Or(p, nullLiteral)
Review comment:
How about merging these to the existing `SimplifyConditionals` as mentioned by @tanelk above? it will then just be a few more pattern matchings, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r480653280
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(cond, l @ Literal(null, _), FalseLiteral) if !cond.nullable => And(cond, l)
Review comment:
what about the other way around? e.g. `case If(cond, FalseLiteral, l @ Literal(null, _))`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29567: [WIP][SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29567:
URL: https://github.com/apache/spark/pull/29567#issuecomment-682366092
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29567: [SPARK-32721][SQL] Simplify if clauses with null and boolean
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29567:
URL: https://github.com/apache/spark/pull/29567#discussion_r479714579
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -463,6 +463,8 @@ object SimplifyConditionals extends Rule[LogicalPlan] with PredicateHelper {
case If(Literal(null, _), _, falseValue) => falseValue
case If(cond, trueValue, falseValue)
if cond.deterministic && trueValue.semanticEquals(falseValue) => trueValue
+ case If(p, l @ Literal(null, _), FalseLiteral) if !p.nullable => And(p, l)
+ case If(p, l @ Literal(null, _), TrueLiteral) if !p.nullable => Or(Not(p), l)
Review comment:
Thanks for being patient with me. I think this new pattern looks okay. And it seems missing in current expression optimization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org