[GitHub] [spark] rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown

[GitBox <gi...@apache.org>]

rdblue commented on a change in pull request #25955: [SPARK-29277][SQL] Add early DSv2 filter and projection pushdown
URL: https://github.com/apache/spark/pull/25955#discussion_r335211563
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala
 ##########
 @@ -55,7 +56,52 @@ case class DataSourceV2Relation(
   }
 
   override def computeStats(): Statistics = {
-    val scan = newScanBuilder().build()
+    if (Utils.isTesting) {
+      // when testing, throw an exception if this computeStats method is called because stats should
+      // not be accessed before pushing the projection and filters to create a scan. otherwise, the
+      // stats are not accurate because they are based on a full table scan of all columns.
+      throw new UnsupportedOperationException(
 
 Review comment:
   Fixed

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org