You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/11/03 13:43:36 UTC
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #8539: ARROW-10462: [CI][Python] Run fsspec tests on Windows
jorisvandenbossche commented on a change in pull request #8539:
URL: https://github.com/apache/arrow/pull/8539#discussion_r516047581
##########
File path: python/pyarrow/filesystem.py
##########
@@ -446,10 +446,6 @@ def _ensure_filesystem(fs):
for mro in inspect.getmro(fs_type):
if mro.__name__ == 'S3FileSystem':
return S3FSWrapper(fs)
- # In case its a simple LocalFileSystem (e.g. dask) use native arrow
- # FS
- elif mro.__name__ == 'LocalFileSystem':
- return LocalFileSystem._get_instance()
Review comment:
The reason this now gives troubles (while this code was already there for more than 2 years), is because with pyarrow >= 2.0, `fsspec` will no longer inherit from pyarrow.filesystem.FileSystem, and so this path was only taken if the passed filesystem was *not* a pyarrow subclass (so before, in pyarrow.fs, we simply didn't take this code path for fsspec filesystems)
##########
File path: python/pyarrow/filesystem.py
##########
@@ -446,10 +446,6 @@ def _ensure_filesystem(fs):
for mro in inspect.getmro(fs_type):
if mro.__name__ == 'S3FileSystem':
return S3FSWrapper(fs)
- # In case its a simple LocalFileSystem (e.g. dask) use native arrow
- # FS
- elif mro.__name__ == 'LocalFileSystem':
- return LocalFileSystem._get_instance()
Review comment:
I *think* it is fine to simply remove this code, as in practice this was also never run anymore with previous pyarrow releases. I suppose this was only ever reached with old fsspec versions that did not yet subclass pyarrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org