[GitHub] spark pull request #22141: [SPARK-25154][SQL] Support NOT IN sub-queries ins...

[gatorsmile <gi...@git.apache.org>]

Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22141#discussion_r211788303
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/subquery.scala ---
    @@ -137,13 +137,21 @@ object RewritePredicateSubquery extends Rule[LogicalPlan] with PredicateHelper {
           plan: LogicalPlan): (Option[Expression], LogicalPlan) = {
         var newPlan = plan
         val newExprs = exprs.map { e =>
    -      e transformUp {
    +      e transformDown {
             case Exists(sub, conditions, _) =>
               val exists = AttributeReference("exists", BooleanType, nullable = false)()
               // Deduplicate conflicting attributes if any.
               newPlan = dedupJoin(
                 Join(newPlan, sub, ExistenceJoin(exists), conditions.reduceLeftOption(And)))
               exists
    +        case (Not(InSubquery(values, ListQuery(sub, conditions, _, _)))) =>
    +          val exists = AttributeReference("exists", BooleanType, nullable = false)()
    +          val inConditions = values.zip(sub.output).map(EqualTo.tupled)
    +          val nullAwareJoinConds = inConditions.map(c => Or(c, IsNull(c)))
    --- End diff --
    
    This sounds a reasonable fix. cc @hvanhovell @liwensun 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org