You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Dorjee Tsering (Jira)" <ji...@apache.org> on 2025/12/24 20:46:00 UTC

[jira] [Updated] (SPARK-54838) Optimize spark partition size

     [ https://issues.apache.org/jira/browse/SPARK-54838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dorjee Tsering updated SPARK-54838:
-----------------------------------
    Issue Type: New Feature  (was: Bug)

> Optimize spark partition size
> -----------------------------
>
>                 Key: SPARK-54838
>                 URL: https://issues.apache.org/jira/browse/SPARK-54838
>             Project: Spark
>          Issue Type: New Feature
>          Components: Spark Core
>    Affects Versions: 4.2.0
>            Reporter: Dorjee Tsering
>            Priority: Minor
>
> I am proposing to add a new function in Dataset class to fix small file problem. We have noticed that if the source data our spark job reads have many small files (KB size), it creates lot of partitions. This PR adds a new function named optimizePartition which when used creates partitions of size 128MB if no desired partition's size passed. You can pass your own desired partition size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org