[jira] [Commented] (SPARK-1860) Standalone Worker cleanup should not clean up running applications

["Mingyu Kim (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047074#comment-14047074 ] 

Mingyu Kim commented on SPARK-1860:
-----------------------------------

[~pwendell], would there be an easy way to tell from the worker node whether an app directory is active or not? In other words, can a worker node get the list of active application ids from the master? I thought this was not doable, so was just going to wipe out all app directories that haven't been used (i.e. no jobs have run even if the the application is still alive) based on the last modified date of the log files. What do you think?

> Standalone Worker cleanup should not clean up running applications
> ------------------------------------------------------------------
>
>                 Key: SPARK-1860
>                 URL: https://issues.apache.org/jira/browse/SPARK-1860
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 1.0.0
>            Reporter: Aaron Davidson
>            Priority: Critical
>             Fix For: 1.1.0
>
>
> The default values of the standalone worker cleanup code cleanup all application data every 7 days. This includes jars that were added to any applications that happen to be running for longer than 7 days, hitting streaming jobs especially hard.
> Applications should not be cleaned up if they're still running. Until then, this behavior should not be enabled by default.



--
This message was sent by Atlassian JIRA
(v6.2#6252)