You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Wes McKinney (JIRA)" <ji...@apache.org> on 2019/06/07 19:25:00 UTC
[jira] [Resolved] (ARROW-4723) Skip _files when reading a directory
containing parquet files
[ https://issues.apache.org/jira/browse/ARROW-4723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wes McKinney resolved ARROW-4723.
---------------------------------
Resolution: Fixed
Fix Version/s: 0.14.0
Issue resolved by pull request 4478
[https://github.com/apache/arrow/pull/4478]
> Skip _files when reading a directory containing parquet files
> -------------------------------------------------------------
>
> Key: ARROW-4723
> URL: https://issues.apache.org/jira/browse/ARROW-4723
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Hossein Falaki
> Priority: Major
> Labels: parquet, pull-request-available
> Fix For: 0.14.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> It is common for Apache Spark or other big data platforms to save additional meta-data files denoted with _ when saving parquet data.
> When using {{make_batch_reader}} to load a directory saved by parquet containing such files we encounter the following error:
> {code:java}
> PetastormMetadataError Traceback (most recent call last)
> /databricks/python/lib/python3.6/site-packages/petastorm/etl/dataset_metadata.py in infer_or_load_unischema(dataset)
> 388 try:
> --> 389 return get_schema(dataset)
> 390 except PetastormMetadataError:
> /databricks/python/lib/python3.6/site-packages/petastorm/etl/dataset_metadata.py in get_schema(dataset)
> 342 raise PetastormMetadataError(
> --> 343 'Could not find _common_metadata file. Use materialize_dataset(..) in'
> 344 ' petastorm.etl.dataset_metadata.py to generate this file in your ETL code.'
> PetastormMetadataError: Could not find _common_metadata file. Use materialize_dataset(..) in petastorm.etl.dataset_metadata.py to generate this file in your ETL code. You can generate it on an existing dataset using petastorm-generate-metadata.py{code}
>
> This is because our Runtime stores the following two files at the end of the job:
> {code:java}
> dbfs:/tmp/petastorm/_committed_4686077819843716563 _committed_4686077819843716563 1965
> dbfs:/tmp/petastorm/_started_4686077819843716563{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)