You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2020/04/14 13:06:00 UTC
[jira] [Created] (ARROW-8446) [Python][Dataset] Detect and use
_metadata file in a list of file paths
Joris Van den Bossche created ARROW-8446:
--------------------------------------------
Summary: [Python][Dataset] Detect and use _metadata file in a list of file paths
Key: ARROW-8446
URL: https://issues.apache.org/jira/browse/ARROW-8446
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Joris Van den Bossche
From https://github.com/dask/dask/pull/6047#discussion_r402391318
When specifying a directory to {{ParquetDataset}}, we will detect if a {{_metadata}} file is present in the directory and use that to populate the {{metadata}} attribute (and not include this file in the list of "pieces", since it does not include any data).
However, when passing a list of files to {{ParquetDataset}}, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).
We _could_ detect it in a list of paths as well.
Note, I mentioned {{ParquetDataset}}, but if working on this, we should probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)