You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/05/18 14:42:00 UTC

[jira] [Updated] (SPARK-7705) Cleanup of .sparkStaging directory fails if application is killed

     [ https://issues.apache.org/jira/browse/SPARK-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-7705:
-----------------------------
      Priority: Minor  (was: Major)
    Issue Type: Improvement  (was: Bug)

Minor since this just amounts to leaving around directories.

> Cleanup of .sparkStaging directory fails if application is killed
> -----------------------------------------------------------------
>
>                 Key: SPARK-7705
>                 URL: https://issues.apache.org/jira/browse/SPARK-7705
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 1.3.0
>            Reporter: Wilfred Spiegelenburg
>            Priority: Minor
>
> When a streaming application is killed while running on YARN the .sparkStaging directory is not cleaned up. Setting spark.yarn.preserve.staging.files=false does not help and still leaves the files around.
> The changes in SPARK-7503 do not catch this case since there is no exception in the shutdown. When the application gets killed the AM gets told to shutdown and the shutdown hook is run but the clean up is not triggered.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org