[jira] [Comment Edited] (ARROW-1967) Python: AssertionError w.r.t Pandas conversion on Parquet files in 0.8.0 dev version

["Wes McKinney (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16311521#comment-16311521 ] 

Wes McKinney edited comment on ARROW-1967 at 1/4/18 3:39 PM:
-------------------------------------------------------------

I guess the pandas metadata changed too many times between 0.7.1 and 0.8.0. I am not sure how practical it will be to support all incarnations of the metadata including in-development ones -- how would you prefer this be handled so you can at least access your data? 


was (Author: wesmckinn):
I guess the pandas metadata changes too many times between 0.7.1 and 0.8.0. I am not sure how practical it will be to support all incarnations of the metadata including in-development ones -- how would you prefer this be handled so you can at least access your data? 

> Python: AssertionError w.r.t Pandas conversion on Parquet files in 0.8.0 dev version
> ------------------------------------------------------------------------------------
>
>                 Key: ARROW-1967
>                 URL: https://issues.apache.org/jira/browse/ARROW-1967
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>            Reporter: Uwe L. Korn
>             Fix For: 0.9.0
>
>
> I have some Parquet files generated a few commit before 0.8.0 that I cannot load in 0.8.0.
> They have the following Pandas metadata:
> {code}
> {u'pandas_version': u'0.20.3', u'index_columns': [u'__index_level_0__'], u'columns': [{u'pandas_type': u'unicode', u'name': u'col1', u'numpy_type': u'object', u'metadata': None}, {u'pandas_type': u'list[unicode]', u'name': u'partition', u'numpy_type': u'object', u'metadata': None}, {u'pandas_type': u'int64', u'name': None, u'numpy_type': u'int64', u'metadata': None}], u'column_indexes': [{u'pandas_type': u'unicode', u'name': None, u'numpy_type': u'object', u'metadata': None}]}
> {code}
> And I get the following exception on loading the file:
> {code}
>     df = table.to_pandas()
>   File "pyarrow/table.pxi", line 1043, in pyarrow.lib.Table.to_pandas (/home/jenkins/.conan/data/pyarrow/0.8.0/by/stable/build/0fda246af0df114df08c06ca4fdd2cac7d7696fd/arrow/python/build/temp.linux-x86_64-2.7/lib.cxx:46331)
>   File "/io/venv_deb8/local/lib/python2.7/site-packages/pyarrow/pandas_compat.py", line 498, in table_to_blockmanager
>     assert len(logical_index_names) == len(index_columns_set)
> AssertionError
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)