[GitHub] [iceberg] samredai commented on pull request #5332: [Python] FsspecFileIO, a FileIO that wraps any fsspec compliant filesystem

[GitBox <gi...@apache.org>]

samredai commented on PR #5332:
URL: https://github.com/apache/iceberg/pull/5332#issuecomment-1228070278

   @Fokko @rdblue I've updated this to `FsspecFileIO` that uses s3fs for locations with an s3 scheme. A couple questions I have is:
   1. The class caches the first filesystem it initializes when a new input file or output file is requested using that uri. Do we want to handle a situation where a single FsspecFileIO instance is handling locations of varying schemes?
   2. In order to use the `Dict[str, str]` type that's been added to the parent FileIO interface, I've set a `client_` prefix for client kwargs and `fs_` prefix for filesystem kwargs. They're then plucked out in the `_fs_properties()` and `_client_properties()` methods. Should this be where we also look for the java property keys and plug them in? For example check for `s3.endpoint` and plug that into the client kwargs as `endpoint_url`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org