[GitHub] [spark] steveloughran commented on pull request #30019: [SPARK-33135][CORE] Use listLocatedStatus from FileSystem implementations

[GitBox <gi...@apache.org>]

steveloughran commented on pull request #30019:
URL: https://github.com/apache/spark/pull/30019#issuecomment-714490460


   >> The new openFile() API will let the caller specify seek policy (sequential, random, adaptive,...) and, if you pass in the known file length, skip the HEAD. It's also an async operation on S3A, even when a HEAD is needed.
   
   > Cool. This is in 3.3.0+ though. We can perhaps explore this once Spark made the switch.
   
   if someone wants to do the work, we could backport the API (with no optimised implementations) to 3.2.x. Same for other public FileSystem API changes. Here work:== backport API plus subset of tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org