You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/21 04:28:17 UTC
[GitHub] [spark] kazuyukitanimura opened a new pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
kazuyukitanimura opened a new pull request #34055:
URL: https://github.com/apache/spark/pull/34055
### What changes were proposed in this pull request?
This PR proposes to improve simplifications of `EqualTo/EqualNullSafe` binary comparators when one side is a boolean literal.
For example: `EqualTo(predicate, TrueLiteral) => predicate`, `EqualNullSafe(predicate, TrueLiteral) => And(predicate, IsNotNull(predicate))`
This PR helps pushing down the filter and reducing unnecessary IO.
### Why are the changes needed?
The following query does not push down the filter in the current implementation
```
SELECT * FROM t WHERE (a AND b) = true
```
although the following equivalent query pushes down the filter as expected.
```
SELECT * FROM t WHERE (a AND b)
```
That is because the first query creates `EqualTo(And(a, b), TrueLiteral)` that is simply not in the form that we can push down. However, we should be able to get it simplified to `And(a, b)`
It is fair for Spark SQL users to expect `(a AND b) = true` performs the same as `(a AND b)`.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added unit tests
```
build/sbt "testOnly *BooleanSimplificationSuite -- -z SPARK-36721"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan edited a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
cloud-fan edited a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926010668
> case a EqualTo TrueLiteral if a.dataType.isInstanceOf[BooleanType]
This is not necessary. The analyzer guarantees that the inputs of binary comparisons are of the same type, and optimizer will never hit an unresolved plan. We can just write `case a EqualTo TrueLiteral`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925753836
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143543/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925341497
@viirya it would be great if you could take another look when you get a chance. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r715125418
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BinaryComparisonSimplificationSuite.scala
##########
@@ -194,4 +199,15 @@ class BinaryComparisonSimplificationSuite extends PlanTest with PredicateHelper
Optimize.execute(testRelation.select(Coalesce(Seq('b, 'd)).as("out")).analyze),
testRelation.select(Coalesce(Seq('b, 'd)).as("out")).analyze)
}
+
+ test("SPARK-36721: Simplify boolean equalities if one side is literal") {
+ checkCondition(boolRelation, And('a, 'b) === TrueLiteral, And('a, 'b))
+ checkCondition(boolRelation, TrueLiteral === And('a, 'b), And('a, 'b))
+ checkCondition(boolRelation, And('a, 'b) === FalseLiteral, Or(Not('a), Not('b)))
+ checkCondition(boolRelation, FalseLiteral === And('a, 'b), Or(Not('a), Not('b)))
+ checkCondition(boolRelation, IsNull('a) <=> TrueLiteral, IsNull('a))
Review comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926041049
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48065/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926203997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924498969
**[Test build #143493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143493/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714908721
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
Review comment:
Does `a` have to be a `Predicate`? It seems like we can apply this optimization to any expression, like a column, or a function that returns boolean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926009823
**[Test build #143556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143556/testReport)** for PR 34055 at commit [`f4976bb`](https://github.com/apache/spark/commit/f4976bbc1dbf766b6e019d9a607a9b7e7f7690fa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925655507
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923634394
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47982/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926084127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48065/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924536609
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926816436
Thanks all! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926010668
> case a EqualTo TrueLiteral if a.dataType.isInstanceOf[BooleanType]
This is not necessary. The analyzer guarantees that the inputs of binary comparisons are of the same type. We can just write `case a EqualTo TrueLiteral`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923656155
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47982/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924541648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926167480
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926009823
**[Test build #143556 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143556/testReport)** for PR 34055 at commit [`f4976bb`](https://github.com/apache/spark/commit/f4976bbc1dbf766b6e019d9a607a9b7e7f7690fa).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714977188
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
Review comment:
I guess the idea is for push down benefit. But for other expressions, it should be also okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714972865
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
+ case TrueLiteral EqualTo (b: Predicate) => b
+ case (a: Predicate) EqualTo FalseLiteral => Not(a)
+ case FalseLiteral EqualTo (b: Predicate) => Not(b)
+ case (a: Predicate) EqualNullSafe TrueLiteral => And(a, IsNotNull(a))
Review comment:
Thank you all, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923824361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143471/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923625462
**[Test build #143471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143471/testReport)** for PR 34055 at commit [`578bdce`](https://github.com/apache/spark/commit/578bdcebaf8ba27e5070ec9eb398ed516b637621).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926127643
**[Test build #143568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143568/testReport)** for PR 34055 at commit [`58c9f7c`](https://github.com/apache/spark/commit/58c9f7cca75c0ed9d398e92f3e136d0976279b64).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sunchao commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
sunchao commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r715060070
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BinaryComparisonSimplificationSuite.scala
##########
@@ -194,4 +199,15 @@ class BinaryComparisonSimplificationSuite extends PlanTest with PredicateHelper
Optimize.execute(testRelation.select(Coalesce(Seq('b, 'd)).as("out")).analyze),
testRelation.select(Coalesce(Seq('b, 'd)).as("out")).analyze)
}
+
+ test("SPARK-36721: Simplify boolean equalities if one side is literal") {
+ checkCondition(boolRelation, And('a, 'b) === TrueLiteral, And('a, 'b))
+ checkCondition(boolRelation, TrueLiteral === And('a, 'b), And('a, 'b))
+ checkCondition(boolRelation, And('a, 'b) === FalseLiteral, Or(Not('a), Not('b)))
+ checkCondition(boolRelation, FalseLiteral === And('a, 'b), Or(Not('a), Not('b)))
+ checkCondition(boolRelation, IsNull('a) <=> TrueLiteral, IsNull('a))
Review comment:
maybe ideally test the case when LHS is nullable but this is just a nit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924541648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925563880
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714529104
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
+ case TrueLiteral EqualTo (b: Predicate) => b
+ case (a: Predicate) EqualTo FalseLiteral => Not(a)
+ case FalseLiteral EqualTo (b: Predicate) => Not(b)
+ case (a: Predicate) EqualNullSafe TrueLiteral => And(a, IsNotNull(a))
Review comment:
Yea, deterministics is an issue. Performance maybe too, although we usually do subexpression elimination, but it possibly adds a bit extra cost. I think best practice should be not duplicate the evaluation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r713515920
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
+ case TrueLiteral EqualTo (b: Predicate) => b
+ case (a: Predicate) EqualTo FalseLiteral => Not(a)
+ case FalseLiteral EqualTo (b: Predicate) => Not(b)
+ case (a: Predicate) EqualNullSafe TrueLiteral => And(a, IsNotNull(a))
Review comment:
Thanks. I will change this to only for the case `!a.nullable`. I originally thought it is ok to evaluate twice in exchange of pushing down the filter. With the restriction of `!a.nullable`, the evaluation will be one time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926126873
> > case a EqualTo TrueLiteral if a.dataType.isInstanceOf[BooleanType]
>
> This is not necessary. The analyzer guarantees that the inputs of binary comparisons are of the same type, and optimizer will never hit an unresolved plan. We can just write `case a EqualTo TrueLiteral`
Thanks, updated @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923824361
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143471/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926275525
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143568/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925671395
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925564684
**[Test build #143543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143543/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714992083
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
Review comment:
Thanks, yes the intention is for pushing down the filter. I will add a comment.
Actually, my original attempt, which I later changed to `Predicate` only, was `case a EqualTo TrueLiteral if a.dataType.isInstanceOf[BooleanType]`. I found that is too disruptive for some tests. I will keep it only for `Predicate` for now, and we can still get the push down benefit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714908721
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
Review comment:
Does `a` have to be a `Predicate`? It seems like we can apply this optimization to any expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926178287
**[Test build #143556 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143556/testReport)** for PR 34055 at commit [`f4976bb`](https://github.com/apache/spark/commit/f4976bbc1dbf766b6e019d9a607a9b7e7f7690fa).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya closed pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya closed pull request #34055:
URL: https://github.com/apache/spark/pull/34055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbtsai commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
dbtsai commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r714515398
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
+ case TrueLiteral EqualTo (b: Predicate) => b
+ case (a: Predicate) EqualTo FalseLiteral => Not(a)
+ case FalseLiteral EqualTo (b: Predicate) => Not(b)
+ case (a: Predicate) EqualNullSafe TrueLiteral => And(a, IsNotNull(a))
Review comment:
Let's say if an expression is not non-deterministic, evaluating it twice could return different results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926179272
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143556/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926275525
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143568/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926203976
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r715000943
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925671395
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925564684
**[Test build #143543 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143543/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] kazuyukitanimura commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
kazuyukitanimura commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923614250
cc @cloud-fan @sunchao @viirya @dbtsai
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925752394
**[Test build #143543 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143543/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926203997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48077/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925624293
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48052/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #34055:
URL: https://github.com/apache/spark/pull/34055#discussion_r712733292
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
##########
@@ -488,6 +489,16 @@ object SimplifyBinaryComparison
// False with inequality
case a GreaterThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
case a LessThan b if canSimplifyComparison(a, b, notNullExpressions) => FalseLiteral
+
+ // One side is Literal
+ case (a: Predicate) EqualTo TrueLiteral => a
+ case TrueLiteral EqualTo (b: Predicate) => b
+ case (a: Predicate) EqualTo FalseLiteral => Not(a)
+ case FalseLiteral EqualTo (b: Predicate) => Not(b)
+ case (a: Predicate) EqualNullSafe TrueLiteral => And(a, IsNotNull(a))
Review comment:
Hmm, doesn't it also mean the predicate `a` will be evaluated more than once?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-925753836
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143543/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923656111
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47982/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926819276
+1, LGTM, too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926179272
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143556/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924498969
**[Test build #143493 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143493/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924514959
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48004/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923811216
**[Test build #143471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143471/testReport)** for PR 34055 at commit [`578bdce`](https://github.com/apache/spark/commit/578bdcebaf8ba27e5070ec9eb398ed516b637621).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-924536036
**[Test build #143493 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143493/testReport)** for PR 34055 at commit [`2e44a5c`](https://github.com/apache/spark/commit/2e44a5c51b3c21edc1a4735ba3a45d55fc1e509e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923625462
**[Test build #143471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143471/testReport)** for PR 34055 at commit [`578bdce`](https://github.com/apache/spark/commit/578bdcebaf8ba27e5070ec9eb398ed516b637621).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926084073
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/48065/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926084127
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/48065/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926127643
**[Test build #143568 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143568/testReport)** for PR 34055 at commit [`58c9f7c`](https://github.com/apache/spark/commit/58c9f7cca75c0ed9d398e92f3e136d0976279b64).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-926274795
**[Test build #143568 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/143568/testReport)** for PR 34055 at commit [`58c9f7c`](https://github.com/apache/spark/commit/58c9f7cca75c0ed9d398e92f3e136d0976279b64).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34055: [SPARK-36721][SQL] Simplify boolean equalities if one side is literal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34055:
URL: https://github.com/apache/spark/pull/34055#issuecomment-923656155
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47982/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org