You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Peter Tian (Jira)" <ji...@apache.org> on 2023/01/09 06:18:00 UTC

[jira] [Created] (SPARK-41946) DiskBlockManager can't clear all of the tmp dirs in k8s

Peter Tian created SPARK-41946:
----------------------------------

             Summary: DiskBlockManager can't clear all of the tmp dirs in k8s
                 Key: SPARK-41946
                 URL: https://issues.apache.org/jira/browse/SPARK-41946
             Project: Spark
          Issue Type: Bug
          Components: Block Manager, Spark Core
    Affects Versions: 3.3.1, 3.2.0
         Environment: Spark 3.2.2

kubernetes v1.20.8 

jdk1.8.0_162
            Reporter: Peter Tian


I have a test for spark on k8s  that my job had completed, but some tmp dir is still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.

The DiskBlockManager may be clear the dirs  just one iteration ， then the process be killed.

The shuffle data will be getting larger and   fill the disk space.

!image-2023-01-09-14-03-01-840.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org