[GitHub] [iceberg] rdblue commented on pull request #3691: Add FileIO, InputFile, and OutputFile abstract base classes

[GitBox <gi...@apache.org>]

rdblue commented on pull request #3691:
URL: https://github.com/apache/iceberg/pull/3691#issuecomment-1016999181


   @emkornfield, yes.
   
   The FileIO abstraction is a weaker abstraction. Iceberg doesn't require file listing, directories, or many of the features of a file system. It only cares that it can read a file as a stream with seek, or write a file and get the byte position from time to time. That's why we maintain a light-weight way to plug in. Having a fsspec FileIO makes a lot of sense for generic use cases, but we don't want to make strict requirements about large dependencies.
   
   This has really helped on the Java side, where we are using this to lower dependence on Hadoop's FileSystem API. And it also allows us to do some custom operations, like hook into S3 setup for non-standard credentials chains.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org