[jira] [Created] (HADOOP-18966) s3a file rename does double HEAD or LIST on source fiile/dir

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-18966:
---------------------------------------

             Summary: s3a file rename does double HEAD or LIST on source fiile/dir
                 Key: HADOOP-18966
                 URL: https://issues.apache.org/jira/browse/HADOOP-18966
             Project: Hadoop Common
          Issue Type: Sub-task
          Components: fs/s3
    Affects Versions: 3.3.6
            Reporter: Steve Loughran


going to add this as a v2 issue, though really it's been latent.

when you do a rename() we do a getFileStatus call to get the list/file status..if this is a dir that's a single LIST; a file then it'll be LIST + HEAD.

when the actual copy is kicked off, 
* if the source is a dir, then a duplicate LIST is initiated straight afterwards.
* FIle: there's a second HEAD to get the metadata for the copy

Proposed.
* LIST initiates a full list, and if it returns a list of objects, that list iterator is passed in to rename.
* full result of HEAD preserved and passed down to copyFile()

will cut out one round trip regardless of source type





--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org