You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2015/06/25 04:15:04 UTC

[jira] [Created] (SPARK-8617) Handle history files better

Thomas Graves created SPARK-8617:
------------------------------------

             Summary: Handle history files better
                 Key: SPARK-8617
                 URL: https://issues.apache.org/jira/browse/SPARK-8617
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 1.4.0
            Reporter: Thomas Graves


Right now there are many conditions in which history files get left around with the .inprogress extension.  The cleaner doesn't remove these because it can't distinguish between something running and left over abandoned files. 

There are cases its really hard to not leave these around (application killed/crashes/etc). But we should still try to handle the history files better. 
  
For instance we could first write them to the .sparkStaging directory and then moving to the history file location before the application finishes.   That way you only get completed history files.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org