[GitHub] [spark] jackylee-ch commented on a change in pull request #35669: [SPARK-38041][SQL] DataFilter pushed down dynamically

[GitBox <gi...@apache.org>]

jackylee-ch commented on a change in pull request #35669:
URL: https://github.com/apache/spark/pull/35669#discussion_r829668943



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -849,6 +849,17 @@ object SQLConf {
     .booleanConf
     .createWithDefault(true)
 
+  val PARQUET_FILTER_DYNAMICALLY_PUSHDOWN_ENABLED =
+    buildConf("spark.sql.parquet.filterPushdown.dynamically")
+      .doc("If true, enables Parquet data filter push-down optimization dynamically. " +
+        "Data filter will be pushed down to ParquetScan with partition Filters, " +
+        "and prune partition filters before pushing to parquet at runtime. " +
+        s"This configuration only has an effect when '${PARQUET_FILTER_PUSHDOWN_ENABLED.key}' " +
+        "is enabled.")
+      .version("3.3.0")

Review comment:
       @huaxingao @cloud-fan @MaxGekk can this pr merged in 3.3.0? If not, I should change the jira.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org