[GitHub] spark pull request #17491: [SPARK-20175][SQL] Exists should not be evaluated...

[nsyca <gi...@git.apache.org>]

Github user nsyca commented on a diff in the pull request:

    https://github.com/apache/spark/pull/17491#discussion_r109181709
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala ---
    @@ -498,3 +498,31 @@ object RewriteCorrelatedScalarSubquery extends Rule[LogicalPlan] {
           }
       }
     }
    +
    +/**
    + * This rule rewrites a EXISTS predicate sub-queries into an Aggregate with count.
    + * So it doesn't be converted to a JOIN later.
    + */
    +object RewriteEmptyExists extends Rule[LogicalPlan] with PredicateHelper {
    +  private def containsAgg(plan: LogicalPlan): Boolean = {
    +    plan.collect {
    +      case a: Aggregate => a
    +    }.nonEmpty
    +  }
    +
    +  def apply(plan: LogicalPlan): LogicalPlan = plan transform {
    +    case Filter(condition, child) =>
    +      val (withSubquery, withoutSubquery) =
    +        splitConjunctivePredicates(condition).partition(SubqueryExpression.hasInOrExistsSubquery)
    +      val newWithSubquery = withSubquery.map(_.transform {
    +        case e @ Exists(sub, conditions, exprId) if conditions.isEmpty && !containsAgg(sub) =>
    --- End diff --
    
    @viirya what we should do is to mark the outer join as an "early out" in the run-time join processing. An aggregation is not cheap as it needs to read the entire table to give the (first) answer. An "early out" logic is just like what we currently have in the LeftSemi join where only the first match of the join value is returned (and the subsequent matches are discarded). A LeftSemi join is a special case of an "early out" inner join where the columns of the right table are not permitted in the output of the join.
    
    IMO, a step forward is to implement the "early out" mechanism in all the run-time join operators, nested-loop, sorted-merge, and hash; and inner, left outer, right outer. Then LeftSemi and LeftAnti will just be special cases of one of those operators.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org