[jira] [Resolved] (ARROW-10146) [Python] Parquet metadata to_dict raises attribute error

["Antoine Pitrou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-10146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Antoine Pitrou resolved ARROW-10146.
------------------------------------
    Fix Version/s: 3.0.0
       Resolution: Fixed

Issue resolved by pull request 8861
[https://github.com/apache/arrow/pull/8861]

> [Python] Parquet metadata to_dict raises attribute error
> --------------------------------------------------------
>
>                 Key: ARROW-10146
>                 URL: https://issues.apache.org/jira/browse/ARROW-10146
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Florian Jetter
>            Assignee: Joris Van den Bossche
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 3.0.0
>
>          Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> h2. Description
> When accessing rowgroup metadata and trying to convert it to a dict using the method {{to_dict}} I sometimes receive an Attribute error.
> This can be consistently produced with an empty dataframe (see example below) but I have also seen it already for non empty dataframes. I couldn't track down what makes the non-empty Attribute errors special, therefore the example below.
> h2. Expected behaviour
> I would expect the to_dict to always, consistently return a dictionary with the appropriate metadata and statistics irregardless of the file content.
> h2. Minimal Example
>  
> {code:python}
> import pandas as pd
> import pyarrow as pa
> import pyarrow.parquet as pq
> df = pd.DataFrame({"col": [1]}).head(0)
> table = pa.Table.from_pandas(df)
> buf = pa.BufferOutputStream()
> pq.write_table(table, buf)
> reader = pa.BufferReader(buf.getvalue())
> parquet_file = pq.ParquetFile(reader)
> # Raises Attribute Error
> parquet_file.metadata.to_dict()
> {code}
> h3. Traceback
> {code:java}
> ~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.FileMetaData.to_dict()
> ~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.RowGroupMetaData.to_dict()
> ~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ColumnChunkMetaData.to_dict()
> AttributeError: 'NoneType' object has no attribute 'to_dict'
> {code}
> h3. Versions
> {code:java}
> In [28]: pa.__version__
> Out[28]: '1.0.1'
> In [29]: pd.__version__
> Out[29]: '1.0.5'
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)