[jira] [Closed] (ARROW-8378) [Python] "empty" dtype metadata leads to wrong Parquet column type

["Wes McKinney (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-8378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wes McKinney closed ARROW-8378.
-------------------------------
    Resolution: Not A Problem

What [~jorisvandenbossche] wrote is right -- I missed that {{astype(np.unicode_)}} was coercing {{None}} to {{'None'}}, so this is not something that we're able to "fix" per se. 

> [Python] "empty" dtype metadata leads to wrong Parquet column type
> ------------------------------------------------------------------
>
>                 Key: ARROW-8378
>                 URL: https://issues.apache.org/jira/browse/ARROW-8378
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 0.16.0
>         Environment: Python: 3.7.6
> Pandas: 0.24.1, 0.25.3, 1.0.3
> Pyarrow: 0.16.0
> OS: OSX 10.15.3
>            Reporter: Diego Argueta
>            Priority: Major
>
> Run the following code with Pandas 0.24.x-1.0.x, and PyArrow 0.16.0 on Python 3.7:
> {code:python}
> import pandas as pd
> import numpy as np
> df_1 = pd.DataFrame({'col': [None, None, None]})
> df_1.col = df_1.col.astype(np.unicode_)
> df_1.to_parquet('right.parq', engine='pyarrow')
> series = pd.Series([None, None, None], dtype=np.unicode_)
> df_2 = pd.DataFrame({'col': series})
> df_2.to_parquet('wrong.parq', engine='pyarrow')
> {code}
> Examine the Parquet column type for each file (I use [parquet-tools|https://github.com/wesleypeck/parquet-tools]). {{right.parq}} has the expected UTF-8 string type. {{wrong.parq}} has an {{INT32}}.
> The following metadata is stored in the Parquet files:
> {{right.parq}}
> {code:json}
> {
>   "column_indexes": [],
>   "columns": [
>     {
>       "field_name": "col",
>       "metadata": null,
>       "name": "col",
>       "numpy_type": "object",
>       "pandas_type": "unicode"
>     }
>   ],
>   "index_columns": [],
>   "pandas_version": "0.24.1"
> }
> {code}
> {{wrong.parq}}
> {code:json}
> {
>   "column_indexes": [],
>   "columns": [
>     {
>       "field_name": "col",
>       "metadata": null,
>       "name": "col",
>       "numpy_type": "object",
>       "pandas_type": "empty"
>     }
>   ],
>   "index_columns": [],
>   "pandas_version": "0.24.1"
> }
> {code}
> The difference between the two is that the {{pandas_type}} for the incorrect file is "empty" rather than the expected "unicode". PyArrow misinterprets this and defaults to a 32-bit integer column.
> The incorrect datatype will cause Redshift to reject the file when we try to read it because the column type in the file doesn't match the column type in the database table.
> I originally filed this as a bug in Pandas (see [this ticket|https://github.com/pandas-dev/pandas/issues/25326]) but they punted me over here because the dtype conversion is handled in PyArrow. I'm not sure how you'd handle this here.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)