You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-issues@hadoop.apache.org by "John Liu (Jira)" <ji...@apache.org> on 2020/08/10 21:09:00 UTC

[jira] [Updated] (MAPREDUCE-7292) Use snapshot diff to build file listing when copying to blob storage

     [ https://issues.apache.org/jira/browse/MAPREDUCE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

John Liu updated MAPREDUCE-7292:
--------------------------------
    Attachment: MAPREDUCE-7292.001.patch

> Use snapshot diff to build file listing when copying to blob storage
> --------------------------------------------------------------------
>
>                 Key: MAPREDUCE-7292
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7292
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: distcp
>            Reporter: John Liu
>            Priority: Minor
>         Attachments: MAPREDUCE-7292.001.patch
>
>
> The DistCp sync option should be extensible for copying to blob storage, which is not a distributed filesystem. Clients of DistCp could benefit from using the HDFS snapshot diff report to create the file listing in less time. A valid use case is to copy new files added to HDFS to a remote blob storage. The client ensures all new files are copied over but does not require the destination filesystem to be a distributed filesystem or have the previous snapshot. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org