[jira] [Commented] (ARROW-2079) [Python][C++] Possibly use `_common_metadata` for schema if `_metadata` isn't available

["Francois Saint-Jacques (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116205#comment-17116205 ] 

Francois Saint-Jacques commented on ARROW-2079:
-----------------------------------------------

Question to users/developers, why the need of 2 files, is it because `_metadata` can be too big?

> [Python][C++] Possibly use `_common_metadata` for schema if `_metadata` isn't available
> ---------------------------------------------------------------------------------------
>
>                 Key: ARROW-2079
>                 URL: https://issues.apache.org/jira/browse/ARROW-2079
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: Python
>            Reporter: Jim Crist
>            Priority: Minor
>              Labels: dataset, dataset-parquet-read, parquet
>
> Currently pyarrow's parquet writer only writes `_common_metadata` and not `_metadata`. From what I understand these are intended to contain the dataset schema but not any row group information.
>  
> A few (possibly naive) questions:
>  
> 1. In the `__init__` for `ParquetDataset`, the following lines exist:
> {code:java}
> if self.metadata_path is not None:
>     with self.fs.open(self.metadata_path) as f:
>         self.common_metadata = ParquetFile(f).metadata
> else:
>     self.common_metadata = None
> {code}
> I believe this should use `common_metadata_path` instead of `metadata_path`, as the latter is never written by `pyarrow`, and is given by the `_metadata` file instead of `_common_metadata` (as seemingly intended?).
>  
> 2. In `validate_schemas` I believe an option should exist for using the schema from `_common_metadata` instead of `_metadata`, as pyarrow currently only writes the former, and as far as I can tell `_common_metadata` does include all the schema information needed.
>  
> Perhaps the logic in `validate_schemas` could be ported over to:
>  
> {code:java}
> if self.schema is not None:
>     pass  # schema explicitly provided
> elif self.metadata is not None:
>     self.schema = self.metadata.schema
> elif self.common_metadata is not None:
>     self.schema = self.common_metadata.schema
> else:
>     self.schema = self.pieces[0].get_metadata(open_file).schema{code}
> If these changes are valid, I'd be happy to submit a PR. It's not 100% clear to me the difference between `_common_metadata` and `_metadata`, but I believe the schema in both should be the same. Figured I'd open this for discussion.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)