You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Tom Augspurger (Jira)" <ji...@apache.org> on 2019/11/08 21:41:00 UTC

[jira] [Created] (ARROW-7102) Make filesystem wrappers compatible with fsspec

Tom Augspurger created ARROW-7102:
-------------------------------------

             Summary: Make filesystem wrappers compatible with fsspec
                 Key: ARROW-7102
                 URL: https://issues.apache.org/jira/browse/ARROW-7102
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
            Reporter: Tom Augspurger


[fsspec|fsspec: https://filesystem-spec.readthedocs.io/en/latest/] defines a common API for a variety filesystem implementations. I'm proposing a FSSpecWrapper, similar to S3FSWrapper, that works with any fsspec implementation.

 

Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to s3fs. [https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320]. This implementation could be removed entirely once an FSSPecWrapper is done, or kept as an alias if it's part of the public API.

 

This is realted to ARROW-3717, which requested a GCSFSWrapper for working with google cloud storage.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)