[jira] [Commented] (SPARK-19167) .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread

["xwebber (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15827704#comment-15827704 ] 

xwebber commented on SPARK-19167:
---------------------------------

 I think this is a core issue that should be fixed. We have meet serveral scenes that this issue affect our application including not able to create new sc in the same process and SPARK_YARN_MODE missing because of the "old" sc.stop haven't finished and set "SparkContext already stopped." too early. I suggest we make sc.stop() blocked until other sc.stop() call finishes anyway.

> .inprogress file can't be renamed if driver exit when OOM occured in AsynchronousListenerBus thread
> ---------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-19167
>                 URL: https://issues.apache.org/jira/browse/SPARK-19167
>             Project: Spark
>          Issue Type: Bug
>            Reporter: xwebber
>
> I was running a spark application and OOM occured in AsynchronousListenerBus thread. This will cause Utils.tryOrStopSparkContext(sparkContext) to do sc.stop(). And then my user code catch the exception that sc is stopping, and this will cause spark driver shut down. but at this time the sc.stop in AsynchronousListenerBus thread hasn't call _cleaner.foreach(_.stop()) yet. This will cause the file left on disk forever. can it be solved?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org