You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/05/11 15:45:00 UTC
[jira] [Created] (ARROW-8766) [Python] A FileSystem implementation
based on Python callbacks
Joris Van den Bossche created ARROW-8766:
--------------------------------------------
Summary: [Python] A FileSystem implementation based on Python callbacks
Key: ARROW-8766
URL: https://issues.apache.org/jira/browse/ARROW-8766
Project: Apache Arrow
Issue Type: Improvement
Components: Python
Reporter: Joris Van den Bossche
The new {{pyarrow.fs}} filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with {{fsspec}} filesystems is therefore also not working anymore.
One possible solution is to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose).
Such a FileSystem implementation would allow to make a {{pyarrow.fs}} wrapper for {{fsspec}} filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)