You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sayed Mohammad Hossein Torabi (Jira)" <ji...@apache.org> on 2023/10/05 10:34:00 UTC

[jira] [Created] (SPARK-45422) Update Partition Stats

Sayed Mohammad Hossein Torabi created SPARK-45422:
-----------------------------------------------------

             Summary: Update Partition Stats
                 Key: SPARK-45422
                 URL: https://issues.apache.org/jira/browse/SPARK-45422
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.5.0
            Reporter: Sayed Mohammad Hossein Torabi


spark brought *spark.sql.statistics.size.autoUpdate.enabled* and it is a good feature for small tables or tables does not contains a lot of files.
It would be great also to introduce a new option that calculates statistics on the partition level. In other words, Instead of altering/updating the whole table statistics, it only gathers the statistics of the partitions that spark writes to the table.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org