You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/04/02 21:05:00 UTC

[jira] [Updated] (ARROW-8245) [Python][Parquet] Skip hidden directories when reading partitioned parquet files

     [ https://issues.apache.org/jira/browse/ARROW-8245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

ASF GitHub Bot updated ARROW-8245:
----------------------------------
    Labels: parquet pull-request-available  (was: parquet)

> [Python][Parquet] Skip hidden directories when reading partitioned parquet files
> --------------------------------------------------------------------------------
>
>                 Key: ARROW-8245
>                 URL: https://issues.apache.org/jira/browse/ARROW-8245
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Caleb Overman
>            Priority: Minor
>              Labels: parquet, pull-request-available
>             Fix For: 0.17.0
>
>
> When writing a partitioned parquet file Spark can create a temporary hidden {{.spark-staging}} directory within the parquet file. Because it is a directory and not a file, it is not skipped when trying to read the parquet file. Pyarrow currently only skips directories prefixed with {{_}}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)