[GitHub] [spark] attilapiros commented on a change in pull request #27453: [SPARK-24914][SQL] Introduce new statistic to improve data size estimate for columnar storage formats (part 1)

[GitBox <gi...@apache.org>]

attilapiros commented on a change in pull request #27453: [SPARK-24914][SQL] Introduce new statistic to improve data size estimate for columnar storage formats (part 1)
URL: https://github.com/apache/spark/pull/27453#discussion_r376480769
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala
 ##########
 @@ -422,14 +425,18 @@ object CatalogTable {
  */
 case class CatalogStatistics(
     sizeInBytes: BigInt,
+    deserFactor: Option[Int] = None,
 
 Review comment:
   It is expected to be a bigger number because of the columnar nature of the underlying file. Moreover it is better to take the ceil of a floating number here as being on the edge risks the an OOM.  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org