[jira] [Updated] (HADOOP-10608) Support appending data in DistCp

["Jing Zhao (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-10608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jing Zhao updated HADOOP-10608:
-------------------------------

    Attachment: HADOOP-10608.000.patch

Very initial patch to demonstrate the idea. The patch includes the following changes:
# add a new FileSystem API to get the file checksum for a given length (i.e., to calculate the checksum of [0, length] part of the file)
# add a new option -append to distcp, which only works with -update and without -skipcrccheck
# Use position read and append to copy the new added data


> Support appending data in DistCp
> --------------------------------
>
>                 Key: HADOOP-10608
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10608
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>         Attachments: HADOOP-10608.000.patch
>
>
> Currently when doing distcp with -update option, for two files with the same file names but with different file length or checksum, we overwrite the whole file. It will be good if we can detect the case where (sourceFile = targetFile + appended_data), and only transfer the appended data segment to the target. This will be very useful if we're doing incremental distcp.



--
This message was sent by Atlassian JIRA
(v6.2#6252)