[jira] [Commented] (SPARK-17547) Temporary shuffle data files may be leaked following exception in write

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491548#comment-15491548 ] 

Apache Spark commented on SPARK-17547:
--------------------------------------

User 'JoshRosen' has created a pull request for this issue:
https://github.com/apache/spark/pull/15104

> Temporary shuffle data files may be leaked following exception in write
> -----------------------------------------------------------------------
>
>                 Key: SPARK-17547
>                 URL: https://issues.apache.org/jira/browse/SPARK-17547
>             Project: Spark
>          Issue Type: Bug
>          Components: Shuffle
>    Affects Versions: 1.5.3, 1.6.0, 2.0.0
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>
> SPARK-8029 modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename the file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle this new temp file.
> This is easy to fix: we just need to add a {{finally}} block to ensure that the temporary file is guaranteed to be either moved or deleted before existing the shuffle write method.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org