[GitHub] [spark] viirya commented on a change in pull request #26751: [SPARK-30107][SQL] Expose nested schema pruning to all V2 sources

[GitBox <gi...@apache.org>]

viirya commented on a change in pull request #26751: [SPARK-30107][SQL] Expose nested schema pruning to all V2 sources
URL: https://github.com/apache/spark/pull/26751#discussion_r354114582
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/parquet/ParquetScanBuilder.scala
 ##########
 @@ -68,6 +68,10 @@ case class ParquetScanBuilder(
   // All filters that can be converted to Parquet are pushed down.
   override def pushedFilters(): Array[Filter] = pushedParquetFilters
 
+  override def pruneColumns(requiredSchema: StructType): Unit = {
+    this.requiredSchema = requiredSchema
+  }
 
 Review comment:
   for a datasource, how does it know passed in requiredSchema is for normal column pruning or nested column pruning?
   
   I have this question because for a datasource that does not support nested column pruning, looks like when SQLConf.get.nestedSchemaPruningEnabled is true, a nested pruning required schema is still passed in. How does such datasource know to act?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org