[GitHub] [spark] Kimahriman commented on a diff in pull request #38003: [SPARK-40565][SQL] Don't push non-deterministic filters to V2 file sources

[GitBox <gi...@apache.org>]

Kimahriman commented on code in PR #38003:
URL: https://github.com/apache/spark/pull/38003#discussion_r980631860


##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala:
##########
@@ -140,6 +140,24 @@ class PruneFileSourcePartitionsSuite extends PrunePartitionSuiteBase with Shared
     }
   }
 
+  test("SPARK-40565: don't push down non-deterministic filters for V2 file sources") {

Review Comment:
   So it looks like that's because of the way the "number of partitions" is calculated for V2 reads at https://github.com/apache/spark/blob/697574c2d5014626d2c960b222b41a8da7c702bf/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/PruneFileSourcePartitionsSuite.scala#L168. It uses `BatchScanExec.partitions.size`, which is the number of RDD partitions, _not_ file partitions. I can see if I can fix that but don't see an easy way like `FileSourceScanExec.selectedPartitions`, there's no intermediate variable to get from a batch scan. Might just have to try to calculate unique partitions values from all files to get actual number of file partitions?
   
   If you change the 2 to 3 for the tests it passes for V1.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org