You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/02 19:15:25 UTC
[GitHub] [spark] Swinky opened a new pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Swinky opened a new pull request #31712:
URL: https://github.com/apache/spark/pull/31712
### What changes were proposed in this pull request?
RewritePredicateSubquery Rule must not update Filters without subqueries.
Following is one such example.
`=== Applying Rule org.apache.spark.sql.catalyst.optimizer.RewritePredicateSubquery ===
Project [a#0] Project [a#0]
!+- Filter (((a#0 > 1) OR (b#1 > 2)) AND ((c#2 > 1) AND (d#3 > 2))) +- Filter ((((a#0 > 1) OR (b#1 > 2)) AND (c#2 > 1)) AND (d#3 > 2))
+- LocalRelation <empty>, [a#0, b#1, c#2, d#3] +- LocalRelation <empty>, [a#0, b#1, c#2, d#3]
`
### Why are the changes needed?
minor change.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing UTs pass.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on a change in pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r588767338
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -98,55 +98,59 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
splitConjunctivePredicates(condition)
.partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)
- // Construct the pruned filter condition.
- val newFilter: LogicalPlan = withoutSubquery match {
- case Nil => child
- case conditions => Filter(conditions.reduce(And), child)
- }
-
- // Filter the plan by applying left semi and left anti joins.
- withSubquery.foldLeft(newFilter) {
- case (p, Exists(sub, conditions, _)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- buildJoin(outerPlan, sub, LeftSemi, joinCond)
- case (p, Not(Exists(sub, conditions, _))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- buildJoin(outerPlan, sub, LeftAnti, joinCond)
- case (p, InSubquery(values, ListQuery(sub, conditions, _, _))) =>
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
- Join(outerPlan, newSub, LeftSemi, joinCond, JoinHint.NONE)
- case (p, Not(InSubquery(values, ListQuery(sub, conditions, _, _)))) =>
- // This is a NULL-aware (left) anti join (NAAJ) e.g. col NOT IN expr
- // Construct the condition. A NULL in one of the conditions is regarded as a positive
- // result; such a row will be filtered out by the Anti-Join operator.
-
- // Note that will almost certainly be planned as a Broadcast Nested Loop join.
- // Use EXISTS if performance matters to you.
+ withSubquery match {
Review comment:
Why not just check if the filter condition has correlated in/exists subquery?
```scala
case f @ Filter(condition, child)
if SubqueryExpression.hasInOrCorrelatedExistsSubquery(condition) =>
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792480053
**[Test build #135853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135853/testReport)** for PR 31712 at commit [`1de70e4`](https://github.com/apache/spark/commit/1de70e4e73c8da2705268f2b6ad7e9d101f7603d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792351177
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789169083
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789453953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135692/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789431790
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Swinky commented on a change in pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
Swinky commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r585854270
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -98,55 +98,59 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
splitConjunctivePredicates(condition)
.partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)
- // Construct the pruned filter condition.
- val newFilter: LogicalPlan = withoutSubquery match {
- case Nil => child
- case conditions => Filter(conditions.reduce(And), child)
Review comment:
@cloud-fan do we want to update `conditions.reduce(And)` as well not create skewed expression trees similar to the review suggestion [here](https://github.com/apache/spark/pull/31318#discussion_r579979610)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792526744
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40435/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789453953
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135692/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792643958
**[Test build #135853 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135853/testReport)** for PR 31712 at commit [`1de70e4`](https://github.com/apache/spark/commit/1de70e4e73c8da2705268f2b6ad7e9d101f7603d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789169083
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792338292
**[Test build #135844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135844/testReport)** for PR 31712 at commit [`126adb8`](https://github.com/apache/spark/commit/126adb89c136df6a90fa9b7a926e2f8fddbdfec7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792338292
**[Test build #135844 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135844/testReport)** for PR 31712 at commit [`126adb8`](https://github.com/apache/spark/commit/126adb89c136df6a90fa9b7a926e2f8fddbdfec7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792353429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Swinky commented on a change in pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
Swinky commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r585854270
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -98,55 +98,59 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
splitConjunctivePredicates(condition)
.partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)
- // Construct the pruned filter condition.
- val newFilter: LogicalPlan = withoutSubquery match {
- case Nil => child
- case conditions => Filter(conditions.reduce(And), child)
Review comment:
@cloud-fan do we want to update `conditions.reduce(And)` as well?
similar to the review suggestion [here](https://github.com/apache/spark/pull/31318#discussion_r579979610)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792596647
thanks, merging to master!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maryannxue commented on a change in pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
maryannxue commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r589585126
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -93,7 +93,8 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
}
def apply(plan: LogicalPlan): LogicalPlan = plan transform {
- case Filter(condition, child) =>
+ case Filter(condition, child)
+ if SubqueryExpression.hasInOrCorrelatedExistsSubquery(condition) =>
Review comment:
Nit: can we pay attention to the indent here? should go 2 spaces further in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #31712:
URL: https://github.com/apache/spark/pull/31712
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r589142505
##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/RewriteSubquerySuite.scala
##########
@@ -67,4 +68,12 @@ class RewriteSubquerySuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+
+ test("SPARK-34598: Filters without subquery must not be modified by RewritePredicateSubquery") {
+ val relation = LocalRelation('a.int, 'b.int, 'c.int, 'd.int)
+ val query = relation.where(('a === 1 || 'b === 2) && ('c === 3 && 'd === 4)).select('a)
+ val tracker = new QueryPlanningTracker
+ Optimize.executeAndTrack(query.analyze, tracker)
+ assert(tracker.rules.get(RewritePredicateSubquery.ruleName).get.numEffectiveInvocations == 0)
Review comment:
nit: `assert(tracker.rules(RewritePredicateSubquery.ruleName).numEffectiveInvocations == 0)`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792345885
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792480053
**[Test build #135853 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135853/testReport)** for PR 31712 at commit [`1de70e4`](https://github.com/apache/spark/commit/1de70e4e73c8da2705268f2b6ad7e9d101f7603d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792645046
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789455672
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40274/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792236388
cc @maryannxue FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789460493
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40274/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792378435
**[Test build #135844 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135844/testReport)** for PR 31712 at commit [`126adb8`](https://github.com/apache/spark/commit/126adb89c136df6a90fa9b7a926e2f8fddbdfec7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792645046
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135853/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792353429
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40426/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792381253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135844/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598][SQL] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792526744
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40435/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789460520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40274/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Swinky commented on a change in pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
Swinky commented on a change in pull request #31712:
URL: https://github.com/apache/spark/pull/31712#discussion_r589069701
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala
##########
@@ -98,55 +98,59 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
splitConjunctivePredicates(condition)
.partition(SubqueryExpression.hasInOrCorrelatedExistsSubquery)
- // Construct the pruned filter condition.
- val newFilter: LogicalPlan = withoutSubquery match {
- case Nil => child
- case conditions => Filter(conditions.reduce(And), child)
- }
-
- // Filter the plan by applying left semi and left anti joins.
- withSubquery.foldLeft(newFilter) {
- case (p, Exists(sub, conditions, _)) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- buildJoin(outerPlan, sub, LeftSemi, joinCond)
- case (p, Not(Exists(sub, conditions, _))) =>
- val (joinCond, outerPlan) = rewriteExistentialExpr(conditions, p)
- buildJoin(outerPlan, sub, LeftAnti, joinCond)
- case (p, InSubquery(values, ListQuery(sub, conditions, _, _))) =>
- // Deduplicate conflicting attributes if any.
- val newSub = dedupSubqueryOnSelfJoin(p, sub, Some(values))
- val inConditions = values.zip(newSub.output).map(EqualTo.tupled)
- val (joinCond, outerPlan) = rewriteExistentialExpr(inConditions ++ conditions, p)
- Join(outerPlan, newSub, LeftSemi, joinCond, JoinHint.NONE)
- case (p, Not(InSubquery(values, ListQuery(sub, conditions, _, _)))) =>
- // This is a NULL-aware (left) anti join (NAAJ) e.g. col NOT IN expr
- // Construct the condition. A NULL in one of the conditions is regarded as a positive
- // result; such a row will be filtered out by the Anti-Join operator.
-
- // Note that will almost certainly be planned as a Broadcast Nested Loop join.
- // Use EXISTS if performance matters to you.
+ withSubquery match {
Review comment:
done, thanks @allisonwang-db
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789432537
Can you add a test to demonstrate the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-792381253
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135844/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31712: [SPARK-34598] RewritePredicateSubquery Rule must not update Filters without subqueries
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31712:
URL: https://github.com/apache/spark/pull/31712#issuecomment-789460520
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40274/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org