[jira] [Resolved] (SPARK-19167) .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-19167.
-------------------------------
    Resolution: Invalid

If you're asking a question -- use user@spark.apache.org. Do you have a suggested change?
I think that in the case of OOM, some bets may be off. Cleanup is best effort. 
You can try calling sc.stop() too because that's good practice, and because it should block until the other stop() call finishes anyway.

> .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19167
>                 URL: https://issues.apache.org/jira/browse/SPARK-19167
>             Project: Spark
>          Issue Type: Bug
>            Reporter: xwebber
>
> I was running a spark application and OOM occured in AsynchronousListenerBus thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do sc.stop(). And then my user code catch the exception that sc is stopping, and this will cause spark driver shut down. but at this time the sc.stop in AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet. This will cause the file left on disk forever. can it be solved?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org