[jira] [Updated] (ARROW-13342) [Python] Categorical boolean column saved as regular boolean in parquet

["Joao Moreira (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-13342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joao Moreira updated ARROW-13342:
---------------------------------
    Summary: [Python] Categorical boolean column saved as regular boolean in parquet  (was: [Python]Categorical boolean column saved as regular boolean in parquet)

> [Python] Categorical boolean column saved as regular boolean in parquet
> -----------------------------------------------------------------------
>
>                 Key: ARROW-13342
>                 URL: https://issues.apache.org/jira/browse/ARROW-13342
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Parquet, Python
>    Affects Versions: 4.0.1
>            Reporter: Joao Moreira
>            Priority: Major
>
> When saving a pandas dataframe to parquet, if there is a categorical column where the categories are boolean, the column is saved as regular boolean.
> This causes an issue because, when reading back the parquet file, I expect the column to still be categorical.
>  
> Reproducible example:
> {code:python}
> import pandas as pd
> import pyarrow
> # Create dataframe with boolean column that is then converted to categorical
> df = pd.DataFrame({'a': [True, True, False, True, False]})
> df['a'] = df['a'].astype('category')
> # Convert to arrow Table and save to disk
> table = pyarrow.Table.from_pandas(df)
> pyarrow.parquet.write_table(table, 'test.parquet')
> # Reload data and convert back to pandas
> table_rel = pyarrow.parquet.read_table('test.parquet')
> df_rel = table_rel.to_pandas()
> {code}
> The arrow {{table}} variable correctly converts the column to an arrow {{DICTIONARY}} type:
> {noformat}
> >>> df['a']
> 0     True
> 1     True
> 2    False
> 3     True
> 4    False
> Name: a, dtype: category
> Categories (2, object): [False, True]
> >>>
> >>> table
> pyarrow.Table
> a: dictionary<values=bool, indices=int8, ordered=0>
> {noformat}
> However, the reloaded column is now a regular boolean:
> {noformat}
> >>> table_rel
> pyarrow.Table
> a: bool
> >>>
> >>> df_rel['a']
> 0     True
> 1     True
> 2    False
> 3     True
> 4    False
> Name: a, dtype: bool
> {noformat}
> I would have expected the column to be read back as categorical.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)