[jira] [Created] (ARROW-10923) Failure to read parquet from s3 after copy of s3-object to new s3-key

["Darren Weber (Jira)" <ji...@apache.org>]

Darren Weber created ARROW-10923:
------------------------------------

             Summary: Failure to read parquet from s3 after copy of s3-object to new s3-key
                 Key: ARROW-10923
                 URL: https://issues.apache.org/jira/browse/ARROW-10923
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Python
            Reporter: Darren Weber


After a parquet file was copied to an s3-bucket and s3-key, pyarrow does not read it from s3.  Desired behavior is that an s3-object for parquet should be self-contained, it should not depend on or track any substantial metadata about the storage engine or file system location it was saved to in such a way that it prevents relocating the object.  To try to replicate the problem, save any parquet file on a linux file system (ext4) and then use the aws-cli to copy that file to any s3-object and then try to use geopandas.read_parquet to load that s3-object.

```
File "/opt/conda/envs/project/lib/python3.7/site-packages/geopandas/io/arrow.py", line 404, in _read_parquet
 table = parquet.read_table(path, columns=columns, **kwargs)
 File "/opt/conda/envs/project/lib/python3.7/site-packages/pyarrow/parquet.py", line 1573, in read_table
 ignore_prefixes=ignore_prefixes,
 File "/opt/conda/envs/project/lib/python3.7/site-packages/pyarrow/parquet.py", line 1434, in __init__
 ignore_prefixes=ignore_prefixes)
 File "/opt/conda/envs/project/lib/python3.7/site-packages/pyarrow/dataset.py", line 667, in dataset
 return _filesystem_dataset(source, **kwargs)
 File "/opt/conda/envs/project/lib/python3.7/site-packages/pyarrow/dataset.py", line 424, in _filesystem_dataset
 fs, paths_or_selector = _ensure_single_source(source, filesystem)
 File "/opt/conda/envs/project/lib/python3.7/site-packages/pyarrow/dataset.py", line 391, in _ensure_single_source
 file_info = filesystem.get_file_info([path])[0]
 File "pyarrow/_fs.pyx", line 429, in pyarrow._fs.FileSystem.get_file_info
 File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
 File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
```




--
This message was sent by Atlassian Jira
(v8.3.4#803005)