[GitHub] [hadoop] steveloughran commented on issue #1591: HADOOP-16629: support copyFile in s3afilesystem

[GitBox <gi...@apache.org>]

steveloughran commented on issue #1591: HADOOP-16629: support copyFile in s3afilesystem
URL: https://github.com/apache/hadoop/pull/1591#issuecomment-541018276
 
 
   Thinking a bit about what a followup patch for cross-store copy would be; I think it'd be how I I think the Multipart Upload API needs to go. There'd be an abstract copier class you'd get an instance of from the dest fs to make 1+ copy under a dest path from a given source
   
   ```
   CopyierBuilder InitiateCopy(Path destination, FileSystem sourceFS, Path source)
   ```
   which you then set ops on to build up the copy
   
   ```
   CopyOperationBuilder builder = copier.copy()
     setSource(sourceStatus) // or a path
     setDest(destPath)
     must("fs.option.overwrite", true)
   ```
   where you could set up things like overwrite, FS permissions, ..
   
   And then kick off the copy
   
   ```
   CompletableFuture<CopyOutcome> outcome = builder.build()
   ```
   
   and await that future. If you are doing many copies, you'd put them in a set of futures and await them all to complete, in whatever order the store chooses. So you don't have to guess what is the optimal order (though a bit of randomisation is always handy)
   
   Like I said: a followup.
   
   What's interesting with that is you could implement a default one which does exec client side in a thread pool. Slower than a rename, but viable

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org