[jira] [Created] (SPARK-52124) Actively Releasing Disk Space After Application Completion in Spark Standalone Mode

["huangsheng (Jira)" <ji...@apache.org>]

huangsheng created SPARK-52124:
----------------------------------

             Summary: Actively Releasing Disk Space After Application Completion in Spark Standalone Mode
                 Key: SPARK-52124
                 URL: https://issues.apache.org/jira/browse/SPARK-52124
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 3.5.5
            Reporter: huangsheng


When submitting tasks using Spark in standalone mode, a folder is generated under the {{work}} directory on each node every time a task is submitted. The naming convention for these folders is, for example, {{{}app-20250212191730-0249{}}}. These folders contain the resource files that each node downloads from the master node when the task is submitted. Although there is a scheduled cleanup mechanism ({{{}spark.worker.cleanup.enabled{}}}), it is not immediate. {color:#FF0000}If a large number of tasks are submitted in a short period of time, and each task depends on a significant amount of external resources, the disk space can be quickly exhausted.{color}
 
Therefore, I suggest actively deleting the disk space occupied under the {{work}} directory after each task is completed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org