You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Roberto Lobo (Jira)" <ji...@apache.org> on 2022/09/06 19:08:00 UTC
[jira] [Commented] (ARROW-17636) Converting Table to pandas raises NotImplementedError (when table previously saved as partitioned parquet dataset)
[ https://issues.apache.org/jira/browse/ARROW-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600947#comment-17600947 ]
Roberto Lobo commented on ARROW-17636:
--------------------------------------
Using an workaround:
{code:java}
conversion_options['types_mapper'] = _TYPE_MAPPINGS.get
try: data = table.to_pandas(**conversion_options) except NotImplementedError: # FIX NotImplemented that happens when partition column is int problematic_columns = [] for tcolumn in list(table.columns): if isinstance(tcolumn.type, pa.DictionaryType): if pa.types.is_integer(tcolumn.type.value_type) and pa.types.is_integer(tcolumn.type.index_type): problematic_columns.append((tcolumn._name, tcolumn, pa.int64()))
for tcolumn_name, tcolumn, tcolumn_type in problematic_columns: table = table.drop([tcolumn_name]) # table = table.append_column(tcolumn_name, [tcolumn.to_numpy()]) table = table.append_column(pa.field(tcolumn._name, tcolumn_type), [tcolumn.to_pylist()]) data = table.to_pandas(**conversion_options)
##############################################################
{code}
> Converting Table to pandas raises NotImplementedError (when table previously saved as partitioned parquet dataset)
> ------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-17636
> URL: https://issues.apache.org/jira/browse/ARROW-17636
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 9.0.0
> Environment: Docker container, based on continuumio/anaconda3
> Python 3.9.12
> PyArrow 9.0.0
> Reporter: Roberto Lobo
> Priority: Major
>
> When converting a table in which one of the column's type is of DictionaryType (values=int32, indices=int32, ordered=0) the conversion to pandas DataFrame fails with:
> NotImplementedError: dictionary<values=int32, indices=int32, ordered=0>
> The dictionary has this conversion not implmented yet.
> This DictionaryType is used as type when using one of the columns (Int64) as one of the parquet's dataset partition columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)