You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Chen He (JIRA)" <ji...@apache.org> on 2016/03/03 00:59:18 UTC

[jira] [Created] (SPARK-13628) Temporary intermediate output file should be renamed before copying to destination filesystem

Chen He created SPARK-13628:
-------------------------------

             Summary: Temporary intermediate output file should be renamed before copying to destination filesystem
                 Key: SPARK-13628
                 URL: https://issues.apache.org/jira/browse/SPARK-13628
             Project: Spark
          Issue Type: Improvement
          Components: Input/Output
    Affects Versions: 1.6.0
            Reporter: Chen He


Spark Executor will dump temporary file into local temp dir, copy it to destination filesystem, and then, rename it. It could be costly for Blobstore (such as openstack swift) which do the actual copy when file is renamed. If it does not affect other components, we may switch the sequence of copy and rename so that Spark can use Blobstore  as final output destination.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org