[jira] [Created] (ARROW-16077) [Python] ArrowInvalid error on reading partitioned parquet files with fsspec.adlfs (pyarrow-7.0.0) due to removed '/' in the ls of path

["Jon Rosenberg (Jira)" <ji...@apache.org>]

Jon Rosenberg created ARROW-16077:
-------------------------------------

             Summary: [Python] ArrowInvalid error on reading partitioned parquet files with fsspec.adlfs (pyarrow-7.0.0) due to removed '/' in the ls of path
                 Key: ARROW-16077
                 URL: https://issues.apache.org/jira/browse/ARROW-16077
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
    Affects Versions: 7.0.0
            Reporter: Jon Rosenberg


Reading a partitioned parquet from adlfs with pyarrow through pandas will throw unnecessary exceptions on not matching forward slashes in the listed files returned from adlfs, ie:

 
{code:python}
import pandas as pd

pd.read_parquet("adl://resource/path/to/parquet/files"){code}
results in exception of the form
{code:bash}
pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path 'path/to/parquet/files/part-0001.parquet', which is outside base dir '/path/to/parquet/files/'{code}
 

and testing with modifying the adlfs method to prepend slashes to all returned files, we still end up with an error on file paths that would otherwise be handled correctly where there is a double slash in a location where there should be one, ie:



 
{code:python}
import pandas as pd

pd.read_parquet("adl://resource/path/to//parquet/files") {code}
would throw
{code:bash}
pyarrow.lib.ArrowInvalid: GetFileInfo() yielded path '/path/to/parquet/files/part-0001.parquet', which is outside base dir '/path/to//parquet/files/' {code}
In both cases the ls has returned correctly from adlfs, given it's discovered the file part-0001.parquet but the pyarrow exception stops what could otherwise be successful processing. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)