[jira] [Created] (HDFS-15522) Use snapshot diff to build file listing when copying to blob storage

["John Liu (Jira)" <ji...@apache.org>]

John Liu created HDFS-15522:
-------------------------------

             Summary: Use snapshot diff to build file listing when copying to blob storage
                 Key: HDFS-15522
                 URL: https://issues.apache.org/jira/browse/HDFS-15522
             Project: Hadoop HDFS
          Issue Type: Improvement
          Components: distcp
            Reporter: John Liu


The DistCp sync option should be extensible for copying to blob storage, which is not a distributed filesystem. Clients of DistCp could benefit from using the HDFS snapshot diff report to create the file listing in less time. A valid use case is to copy new files added to HDFS to a remote blob storage. The client ensures all new files are copied over but does not require the destination filesystem to be a distributed filesystem or have the previous snapshot. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org