[jira] [Updated] (ARROW-8446) [Python][Dataset] Detect and use _metadata file in a list of file paths

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-8446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joris Van den Bossche updated ARROW-8446:
-----------------------------------------
    Labels: dataset dataset-dask-integration parquet  (was: dataset parquet)

> [Python][Dataset] Detect and use _metadata file in a list of file paths
> -----------------------------------------------------------------------
>
>                 Key: ARROW-8446
>                 URL: https://issues.apache.org/jira/browse/ARROW-8446
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Joris Van den Bossche
>            Priority: Major
>              Labels: dataset, dataset-dask-integration, parquet
>
> From https://github.com/dask/dask/pull/6047#discussion_r402391318
> When specifying a directory to {{ParquetDataset}}, we will detect if a {{_metadata}} file is present in the directory and use that to populate the {{metadata}} attribute (and not include this file in the list of "pieces", since it does not include any data).
>  
> However, when passing a list of files to {{ParquetDataset}}, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).
> We _could_ detect it in a list of paths as well.
> Note, I mentioned {{ParquetDataset}}, but if working on this, we should probably directly do it in the datasets API-based version.  
> Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)