You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by "Caleb Overman (Jira)" <ji...@apache.org> on 2020/03/27 15:16:00 UTC

[jira] [Created] (ARROW-8245) [Python] Skip hidden directories when reading partitioned parquet files

Caleb Overman created ARROW-8245:
------------------------------------

             Summary: [Python] Skip hidden directories when reading partitioned parquet files
                 Key: ARROW-8245
                 URL: https://issues.apache.org/jira/browse/ARROW-8245
             Project: Apache Arrow
          Issue Type: Bug
          Components: Python
            Reporter: Caleb Overman


When writing a partitioned parquet file Spark can create a temporary hidden `.spark-staging` directory within the parquet file. Because it is a directory and not a file, it is not skipped when trying to read the parquet file. Pyarrow currently only skips directories prefixed with `_`.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)