[GitHub] [spark] aokolnychyi commented on a change in pull request #30558: [SPARK-33612][SQL] Add v2SourceRewriteRules batch to Optimizer

[GitBox <gi...@apache.org>]

aokolnychyi commented on a change in pull request #30558:
URL: https://github.com/apache/spark/pull/30558#discussion_r533123744



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -189,6 +189,9 @@ abstract class Optimizer(catalogManager: CatalogManager)
     // plan may contain nodes that do not report stats. Anything that uses stats must run after
     // this batch.
     Batch("Early Filter and Projection Push-Down", Once, earlyScanPushDownRules: _*) :+
+    // This batch rewrites plans for v2 tables. It should be run after the operator
+    // optimization batch and before any batches that depend on stats.
+    Batch("V2 Source Rewrite Rules", Once, v2SourceRewriteRules: _*) :+

Review comment:
       Right now, scan construction and pushdown happens in the same method. We could split that later so I agree with moving this batch before early filter and projection pushdown.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org