You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2016/09/14 21:52:20 UTC
[jira] [Created] (SPARK-17547) Temporary shuffle data files may be
leaked following exception in write
Josh Rosen created SPARK-17547:
----------------------------------
Summary: Temporary shuffle data files may be leaked following exception in write
Key: SPARK-17547
URL: https://issues.apache.org/jira/browse/SPARK-17547
Project: Spark
Issue Type: Bug
Components: Shuffle
Affects Versions: 2.0.0, 1.6.0, 1.5.3
Reporter: Josh Rosen
Assignee: Josh Rosen
SPARK-8029 modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename the file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle this new temp file.
This is easy to fix: we just need to add a {{finally}} block to ensure that the temporary file is guaranteed to be either moved or deleted before existing the shuffle write method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org