[GitHub] [hadoop] steveloughran commented on pull request #3736: HADOOP-18028. improve S3 read speed using prefetching & caching

[GitBox <gi...@apache.org>]

steveloughran commented on pull request #3736:
URL: https://github.com/apache/hadoop/pull/3736#issuecomment-1043409450


   I'm not deliberately ignoring you, just so overloaded with my own work that I've not been able to review anything.
   
   we've put mukund's vectored IO patch into a feature branch with the idea being we will still do the normal review process before every patch, but we can get that short chain of patches lined up before the big merge into trunk. we can also play rebase and interactive rebase too.
    
   would that work for you too? so patch 1 is "take what there is", patch 2+ being the changes we want in before shipping
   
   the risk is always it gets forgotten, so we still need to push hard to get it into a state where it can be used as an option, the goal being "no side effects if not used", including nothing extra on the classpath.
   
   
   meanwhile, have a look at this https://github.com/apache/hadoop/pull/2584
   
   its a big patch, but a key feature is you can declare your read policy, with whole-file being an option as well as random, sequential and vectored.
   
   distcp and the CLI tools all declare their read plans this way
   
   i'd like this in before both the vectored io and your stream, so you can use it to help decide whether to cache etc, and to support custom options as well as parse the newly defined standard ones


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org