[jira] [Created] (ARROW-8766) [Python] A FileSystem implementation based on Python callbacks

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

Joris Van den Bossche created ARROW-8766:
--------------------------------------------

             Summary: [Python] A FileSystem implementation based on Python callbacks
                 Key: ARROW-8766
                 URL: https://issues.apache.org/jira/browse/ARROW-8766
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
            Reporter: Joris Van den Bossche


The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with {{fsspec}} filesystems is therefore also not working anymore. 

One possible solution is  to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose). 

Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper for {{fsspec}} filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)