[jira] [Updated] (SPARK-48378) Limit the maximum number of dynamic partitions

["Dongjoon Hyun (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-48378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Dongjoon Hyun updated SPARK-48378:
----------------------------------
    Fix Version/s:     (was: 4.0.0)

> Limit the maximum number of dynamic partitions
> ----------------------------------------------
>
>                 Key: SPARK-48378
>                 URL: https://issues.apache.org/jira/browse/SPARK-48378
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.0.0
>            Reporter: guihuawen
>            Priority: Major
>
> This issues https://issues.apache.org/jira/browse/SPARK-37217  
> ‘Assuming that 1001 partitions are written, the data of 10001 partitions will be deleted first, but because {{hive.exec.max.dynamic.partitions}} is 1000 by default, loadDynamicPartitions will fail at this time, but the data of 1001 partitions has been deleted.
> So we can check whether the number of dynamic partitions is greater than {{hive.exec.max.dynamic.partitions}} before deleting, it should fail quickly at this time.’
>  
> Can this be done to make it more comprehensive.
> If a task on the execution node has already generated partitions that exceed hive. exec. max. dynamic. partitions, the task should be stopped at the execution node because the cost of generating data is high.
> If the executing node does not generate a partition exceeding hive.axec.max.dynamic.partitions for a certain task, it is still determined through the driver.
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org