You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/05/21 05:37:27 UTC

[jira] [Resolved] (SPARK-4287) Add TTL-based cleanup in external shuffle service

     [ https://issues.apache.org/jira/browse/SPARK-4287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon resolved SPARK-4287.
---------------------------------
    Resolution: Incomplete

> Add TTL-based cleanup in external shuffle service
> -------------------------------------------------
>
>                 Key: SPARK-4287
>                 URL: https://issues.apache.org/jira/browse/SPARK-4287
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.2.0
>            Reporter: Andrew Or
>            Priority: Major
>              Labels: bulk-closed
>
> A problem with long-running SparkContexts using the external shuffle service is that its shuffle files may never be cleaned. This is because the ContextCleaner may no longer have executors to go through to clean up these files (they may be killed intentionally). We should have a TTL-based timeout that does this as a backup, defaulting to a timeout of a week or something.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org