[jira] [Commented] (HADOOP-10608) Support incremental data copy in DistCp

["Tsz Wo Nicholas Sze (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14003070#comment-14003070 ] 

Tsz Wo Nicholas Sze commented on HADOOP-10608:
----------------------------------------------

The patch is quite clean and simple!  Some comments:

- In DFSClient.getFileChecksum(..), there is a second callGetBlockLocations(namenode, src, 0, Long.MAX_VALUE).  We should change Long.MAX_VALUE to length.

- FileSystem.getFileChecksum(Path) should call the new FileSystem.getFileChecksum(..) method with length = Long.MAX_VALUE.  FileSystem subclasses such as DistributedFileSystem only have to override the new getFileChecksum(..) method.

- In DataXceiver.calcPartialBlockChecksum(..), toDigest == n since n <= remaining by the read(..) method.  So, we may remove either n or toDigest.

- We should update WebHDFS and other file system subclasses.  It could be done separately.

> Support incremental data copy in DistCp
> ---------------------------------------
>
>                 Key: HADOOP-10608
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10608
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HADOOP-10608.000.patch
>
>
> Currently when doing distcp with -update option, for two files with the same file names but with different file length or checksum, we overwrite the whole file. It will be good if we can detect the case where (sourceFile = targetFile + appended_data), and only transfer the appended data segment to the target. This will be very useful if we're doing incremental distcp.



--
This message was sent by Atlassian JIRA
(v6.2#6252)