[jira] [Updated] (ARROW-14959) [Python] Reading Hive-style partitioned parquet files from GCS

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-14959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche updated ARROW-14959:
------------------------------------------
    Summary: [Python] Reading Hive-style partitioned parquet files from GCS  (was: Reading Hive-style partitioned parquet files from GCS)

> [Python] Reading Hive-style partitioned parquet files from GCS
> --------------------------------------------------------------
>
>                 Key: ARROW-14959
>                 URL: https://issues.apache.org/jira/browse/ARROW-14959
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 6.0.1
>            Reporter: Garrett Weaver
>            Priority: Minor
>
> Trying to read a spark-generated hive-style partitioned parquet dataset with *gcsfs* and {*}pyarrow{*}, but getting a *FileNotFoundError* if I try to read from the base directory or even if try to read directly from one of the partitions. Not sure if I am doing something wrong or it is not supported.
> Note that I have successfully read this hive-style partitioned parquet dataset using other methods to rule out any other issues, including:
>  * Successful read with pyspark by using spark.read.parquet
>  * Successful read of a specific partition by passing list of paths to `ParquetDataset`
>  * Also, tested reading another spark-generated parquet dataset with no Hive-style partitions from GCS and that worked as well.
> Below is what I am trying:
>  
> {code:python}
> import gcsfs
> import pyarrow as pa
>  
> gcs = gcsfs.GCSFileSystem()
>  
> partitions = pa.dataset.partitioning(
>     pa.schema([("partition_var", pa.string())]), dictionaries="infer", flavor="hive"
> )
>  
> partitioned_dataset = pa.dataset.dataset(
>     "path/to/partitioned/dataset/base/dir",
>     filesystem=gcs,
>     format="parquet",
>     partitioning=partitions,
> )
>  
> partition_of_dataset = pa.dataset.dataset(
>     "path/to/partitioned/dataset/base/dir/partition_var=some_value",
>     filesystem=gcs,
>     format="parquet",
> ){code}
>  
> The errors returned for both are below:
> {code:java}
> FileNotFoundError: path/to/partitioned/dataset/base/dir/
> FileNotFoundError: path/to/partitioned/dataset/base/dir/partition_var=some_value/{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)