[GitHub] [spark] steveloughran commented on pull request #29959: [SPARK-32381][CORE][SQL][FOLLOWUP] More cleanup on HadoopFSUtils

[GitBox <gi...@apache.org>]

steveloughran commented on pull request #29959:
URL: https://github.com/apache/spark/pull/29959#issuecomment-707163511


   One thing to know here is that while listStatus etc is slow against object stores, it's not enough of a bandwidth killer than you get much speedup executing it across a spark cluster. Same process-many-threads: yes. Remote exec: no.
   
   I am going to make this class: org.apache.hadoop.mapred.LocatedFileStatusFetcher @Public/Evolving: in this PR: https://github.com/apache/hadoop/pull/2324
    it does do multithread scanning today and is fairly stable. 
   
   It's not the perfect API -I'd have a builder and actually return a remoteIterator of statuses which would block while new listing entries arrived, but it's there. With that PR will also collect aggregate stats on IO operations from any FS whose list calls provide them.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org