[GitHub] [hadoop] bogthe commented on pull request #3101: HADOOP-17139 Re-enable optimized copyFromLocal implementation in S3AFileSystem

[GitBox <gi...@apache.org>]

bogthe commented on pull request #3101:
URL: https://github.com/apache/hadoop/pull/3101#issuecomment-868513275


   **This PR is still in development**
   
   Alright, finally got some time to  implement some of the suggested changes.
   
   What new up until now:
   - `listFilesAndDirs` a new `RemoteIterator` similar to `listFiles` that includes `LocatedFileStatus` for directories too. It's handy when we want to detect empty directories;
   - The new `CopyFromLocalOperation` in S3a which "borrows" ideas from [the Cloudup project](https://github.com/steveloughran/cloudstore/blob/trunk/src/main/java/org/apache/hadoop/fs/tools/cloudup/Cloudup.java);
   
   What's left :
   - Write up the test cases for an `AbstractContractCopyFromLocalTest` class as described above;
   - Update / Add documentation;
   - Do one final "polish" of rough edges;
   
   What was surprising:
   - `trackDurationAndSpan(stat, path, new CopyFromLocalOperation(...))` did create a valid span however the operation class "didn't have access to it" (i.e. any span from inside of `CopyFromLocalOperation` was inactive) hence the `() -> new CopyFromLocalOperation(...).execute()` call.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org