[jira] [Assigned] (SPARK-57993) Support advisory partition size in rebalance hint

["Wenchen Fan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-57993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wenchen Fan reassigned SPARK-57993:
-----------------------------------

    Assignee: Zhen Wang

> Support advisory partition size in rebalance hint
> -------------------------------------------------
>
>                 Key: SPARK-57993
>                 URL: https://issues.apache.org/jira/browse/SPARK-57993
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Zhen Wang
>            Assignee: Zhen Wang
>            Priority: Major
>              Labels: pull-request-available
>
> Spark SQL users often insert a *REBALANCE* hint before writing query results to control small files and produce output partitions with a reasonable target size.
> Today, the target size used by rebalance is controlled by the session-level *spark.sql.adaptive.advisoryPartitionSizeInBytes* configuration. This is too broad for write-side file size control: increasing the advisory partition size for the final write stage can also affect earlier AQE stages, including shuffle coalescing and read/computation parallelism before the final output stage.
> This proposes adding a *REBALANCE_BY_SIZE* hint that works like *REBALANCE* and accepts an optional advisory partition size in bytes as the first parameter, followed by optional partition columns. Users can insert this hint before writing and set the target partition size only for the rebalance shuffle, so the final output file size can be controlled without changing the advisory size used by preceding stages.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org