You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wangyum (via GitHub)" <gi...@apache.org> on 2024/02/29 03:28:54 UTC

[PR] [SPARK-47222][SQL] fileCompressionFactor also applied to the size of the table [spark]

wangyum opened a new pull request, #45329:
URL: https://github.com/apache/spark/pull/45329

   ### What changes were proposed in this pull request?
   
   This PR makes `spark.sql.sources.fileCompressionFactor` also applied to the size of the table.
   
   ### Why are the changes needed?
   
   To keep the behavior consistent. For example:
   ```shell
   bin/spark-shell --conf spark.sql.catalogImplementation=in-memory
   ```
   ```
   scala> spark.range(5).write.parquet("/tmp/spark/parquet")
                                                                                   
   scala> spark.sql("set spark.sql.sources.fileCompressionFactor=2.0")
   res1: org.apache.spark.sql.DataFrame = [key: string, value: string]
   
   scala> spark.sql("create table t(id long) using parquet location '/tmp/spark/parquet'")
   res2: org.apache.spark.sql.DataFrame = []
   
   scala> spark.sql("select * from t").explain("cost")
   == Optimized Logical Plan ==
   Relation spark_catalog.default.t[id#13L] parquet, Statistics(sizeInBytes=5.2 KiB) // The sizeInBytes is location.sizeInBytes * compressionFactor
   
   scala> sql("ANALYZE TABLE t COMPUTE STATISTICS noscan")
   res4: org.apache.spark.sql.DataFrame = []
   
   scala> spark.sql("select * from t").explain("cost")
   == Optimized Logical Plan ==
   Relation spark_catalog.default.t[id#17L] parquet, Statistics(sizeInBytes=2.6 KiB) // The sizeInBytes is table statistics
   ```
   
   After this PR:
   ```
   scala> spark.sql("select * from t").explain("cost")
   == Optimized Logical Plan ==
   Relation spark_catalog.default.t[id#17L] parquet, Statistics(sizeInBytes=5.2 KiB) // The sizeInBytes is table statistics * compressionFactor
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Unit test.
   
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-47222][SQL] fileCompressionFactor also applied to the size of the table [spark]

Posted by "wangyum (via GitHub)" <gi...@apache.org>.
wangyum commented on PR #45329:
URL: https://github.com/apache/spark/pull/45329#issuecomment-1970415466

   cc @cloud-fan 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org