[jira] [Commented] (HADOOP-12046) Avoid creating "._COPYING_" temporary file when copying file to Swift file system

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-12046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14567091#comment-14567091 ] 

Steve Loughran commented on HADOOP-12046:
-----------------------------------------

The reason the marker is there is because copy operations are not atomic in a normal filestore: the partly copied file is visible. In contrast, in an object store, the copy is atomic: the new entry is only visible after the PUT completes. What we can't do, today, is easily distinguish object store from real FS, for code upstream (CLI, distcp, output committers) to act differently.

If you look at HADOOP-9565, we've been discussing offloading the copy operation to the FS implementation itself. as some object stores (S3) implement COPY internally, so can do a copy without the client having to do a full process, or that rename(), which is very expensive


> Avoid creating "._COPYING_" temporary file when copying file to Swift file system
> ---------------------------------------------------------------------------------
>
>                 Key: HADOOP-12046
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12046
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: fs/swift
>    Affects Versions: 2.7.0
>            Reporter: Chen He
>            Assignee: Chen He
>
> When copy file from HDFS or local to another file system implementation, in CommandWithDestination.java, it creates a temp file by adding suffix "._COPYING_". Once file is successfully copied, it will remove the suffix by rename(). 
> try {
>       PathData tempTarget = target.suffix("._COPYING_");
>       targetFs.setWriteChecksum(writeChecksum);
>       targetFs.writeStreamToFile(in, tempTarget, lazyPersist);
>       targetFs.rename(tempTarget, target);
>     } finally {
>       targetFs.close(); // last ditch effort to ensure temp file is removed
>     }
> It is not costly in HDFS. However, if copy to Swift file system, the rename process is to create a new file. It is not efficient if users copy a lot of files to swift file system. I did some tests, for a 1G file copying to swift, it will take 10% more time. We should only do the copy one time for Swift file system. Changes should be limited to the Swift driver level.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)