You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Peter Tian (Jira)" <ji...@apache.org> on 2023/01/11 11:44:00 UTC

[jira] [Updated] (SPARK-41946) DiskBlockManager can't clear all of the tmp dirs in k8s

     [ https://issues.apache.org/jira/browse/SPARK-41946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Tian updated SPARK-41946:
-------------------------------
    Description: 
I have a test for spark on k8s  that my job had completed, but some tmp dirs are still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.

The DiskBlockManager may be clear the dirs  just one iteration ， then the process be killed.

The shuffle data will be getting larger and   fill the disk space.

!image-2023-01-09-14-03-01-840.png!

  was:
I have a test for spark on k8s  that my job had completed, but some tmp dir is still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.

The DiskBlockManager may be clear the dirs  just one iteration ， then the process be killed.

The shuffle data will be getting larger and   fill the disk space.

!image-2023-01-09-14-03-01-840.png!


> DiskBlockManager can't clear all of the tmp dirs in k8s
> -------------------------------------------------------
>
>                 Key: SPARK-41946
>                 URL: https://issues.apache.org/jira/browse/SPARK-41946
>             Project: Spark
>          Issue Type: Bug
>          Components: Block Manager, Spark Core
>    Affects Versions: 3.2.0, 3.3.1
>         Environment: Spark 3.2.2
> kubernetes v1.20.8 
> jdk1.8.0_162
>            Reporter: Peter Tian
>            Priority: Major
>
> I have a test for spark on k8s  that my job had completed, but some tmp dirs are still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is mount on local NVMe SSD.
> The DiskBlockManager may be clear the dirs  just one iteration ， then the process be killed.
> The shuffle data will be getting larger and   fill the disk space.
> !image-2023-01-09-14-03-01-840.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org