[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #14717: ARROW-18272: [Python] Support filesystem parameter in ParquetFile

[GitBox <gi...@apache.org>]

jorisvandenbossche commented on code in PR #14717:
URL: https://github.com/apache/arrow/pull/14717#discussion_r1037948892


##########
python/pyarrow/parquet/core.py:
##########
@@ -304,7 +308,15 @@ def __init__(self, source, *, metadata=None, common_metadata=None,
                  read_dictionary=None, memory_map=False, buffer_size=0,
                  pre_buffer=False, coerce_int96_timestamp_unit=None,
                  decryption_properties=None, thrift_string_size_limit=None,
-                 thrift_container_size_limit=None):
+                 thrift_container_size_limit=None, filesystem=None):
+
+        self._close_source = getattr(source, 'closed', True)
+
+        filesystem, source = _resolve_filesystem_and_path(source, filesystem)

Review Comment:
   When passing a local path as string, this will now always return a filesystem object. I think that means that the `memory_map` option will be ignored (which is otherwise passed to `reader.open(..)`, and there passed to `get_reader`. 
   So we might need to update `_resolve_filesystem_and_path` a bit to pass through the memory_map option (a local filesystem can take that as a param, so it can be passed through).
   
   
   



##########
python/pyarrow/tests/parquet/test_parquet_file.py:
##########
@@ -177,8 +178,12 @@ def test_parquet_file_pass_directory_instead_of_file(tempdir):
     path = tempdir / 'directory'
     os.mkdir(str(path))
 
-    with pytest.raises(IOError, match="Expected file path"):
+    msg = f"Cannot open for reading: path '{str(path)}' is a directory"
+    with pytest.raises(IOError) as exc:
         pq.ParquetFile(path)
+    if exc.errisinstance(PermissionError) and sys.platform == 'win32':
+        return  # Windows CI can get a PermissionError here.

Review Comment:
   cc @pitrou do you have any idea why trying to open a directory on windows gives a PermissionError instead of a normal IO error with the informative message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org