You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2023/01/14 04:34:09 UTC
[GitHub] [spark] cloud-fan commented on a diff in pull request #39131: [SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations
cloud-fan commented on code in PR #39131:
URL: https://github.com/apache/spark/pull/39131#discussion_r1070214768
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushDownLeftSemiAntiJoin.scala:
##########
@@ -61,9 +61,10 @@ object PushDownLeftSemiAntiJoin extends Rule[LogicalPlan]
}
// LeftSemi/LeftAnti over Aggregate, only push down if join can be planned as broadcast join.
- case join @ Join(agg: Aggregate, rightOp, LeftSemiOrAnti(_), _, _)
+ case join @ Join(agg: Aggregate, rightOp, LeftSemiOrAnti(_), joinCond, _)
if agg.aggregateExpressions.forall(_.deterministic) && agg.groupingExpressions.nonEmpty &&
!agg.aggregateExpressions.exists(ScalarSubquery.hasCorrelatedScalarSubquery) &&
+ canPushThroughCondition(agg.children, joinCond, rightOp) &&
Review Comment:
I think we should rewrite the `joinCondition` assuming the join has already been pushed through Aggregate. That said, we need to do alias replacement for `joinCondition` first. cc @EnricoMi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org