[jira] [Updated] (ARROW-8766) [Python] A FileSystem implementation based on Python callbacks

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-8766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche updated ARROW-8766:
-----------------------------------------
        Parent: ARROW-7102
    Issue Type: Sub-task  (was: Improvement)

> [Python] A FileSystem implementation based on Python callbacks
> --------------------------------------------------------------
>
>                 Key: ARROW-8766
>                 URL: https://issues.apache.org/jira/browse/ARROW-8766
>             Project: Apache Arrow
>          Issue Type: Sub-task
>          Components: Python
>            Reporter: Joris Van den Bossche
>            Priority: Major
>
> The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with {{fsspec}} filesystems is therefore also not working anymore. 
> One possible solution is  to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose). 
> Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper for {{fsspec}} filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)