[GitHub] peter-toth commented on a change in pull request #23802: [SPARK-26893][SQL] Allow pushdown of partition pruning subquery filters to file source

[GitBox <gi...@apache.org>]

peter-toth commented on a change in pull request #23802: [SPARK-26893][SQL] Allow pushdown of partition pruning subquery filters to file source
URL: https://github.com/apache/spark/pull/23802#discussion_r257303536
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitions.scala
 ##########
 @@ -39,15 +39,8 @@ private[sql] object PruneFileSourcePartitions extends Rule[LogicalPlan] {
             _,
             _))
         if filters.nonEmpty && fsRelation.partitionSchemaOption.isDefined =>
-      // The attribute name of predicate could be different than the one in schema in case of
-      // case insensitive, we should change them to match the one in schema, so we donot need to
-      // worry about case sensitivity anymore.
-      val normalizedFilters = filters.filterNot(SubqueryExpression.hasSubquery).map { e =>
-        e transform {
-          case a: AttributeReference =>
-            a.withName(logicalRelation.output.find(_.semanticEquals(a)).get.name)
-        }
-      }
+      val normalizedFilters =
 
 Review comment:
   Reviewers please note that this change is not related to this PR actually. But I touched `DataSourceStrategy.normalizeFilters()` so I thought this is it is good occasion to refactor this.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org