You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2022/05/17 09:45:00 UTC

[jira] [Commented] (ARROW-16580) [Python] Multiindex levels order is not preserved after a from_pandas/to_pandas

    [ https://issues.apache.org/jira/browse/ARROW-16580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17538069#comment-17538069 ] 

Joris Van den Bossche commented on ARROW-16580:
-----------------------------------------------

[~Fabien Aulaire] it's certainly the good place to report issue, thanks for the report

You are correct that currently the order of the values in each MultiIndex level is not preserved. 
To help with a faithful rountrip of pandas dataframes, pyarrow stores "pandas" metadata in its schema (see below), but this does not include the level values itself (it only includes the names of the levels, and then the individual column names, from which the MultiIndex level values are reconstructed). And when reconstructing those values, we use {{pd.MultiIndex.from_tuples}} (https://github.com/apache/arrow/blob/52a051b1f01669647d8f4d16d7de30c10265dc51/python/pyarrow/pandas_compat.py#L911), which will sort the values lexicographically, I think.

In theory, we could add more information about the MultiIndex levels in the pandas metadata, but I am not sure we should do that in this case (the levels can in principle be a large array, and we still need to serialize/deserialize it to some string format).


---

For your code example above, the pandas metadata look like:

{code}
In [9]: table.schema.pandas_metadata
Out[9]: 
{'index_columns': [{'kind': 'range',
   'name': None,
   'start': 0,
   'stop': 2,
   'step': 1}],
 'column_indexes': [{'name': 'l1',
   'field_name': 'l1',
   'pandas_type': 'unicode',
   'numpy_type': 'object',
   'metadata': {'encoding': 'UTF-8'}},
  {'name': 'l2',
   'field_name': 'l2',
   'pandas_type': 'unicode',
   'numpy_type': 'object',
   'metadata': {'encoding': 'UTF-8'}}],
 'columns': [{'name': "('foo', 'Toyota')",
   'field_name': "('foo', 'Toyota')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None},
  {'name': "('foo', 'Ford')",
   'field_name': "('foo', 'Ford')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None},
  {'name': "('foo', 'Alfa')",
   'field_name': "('foo', 'Alfa')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None},
  {'name': "('bar', 'Toyota')",
   'field_name': "('bar', 'Toyota')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None},
  {'name': "('bar', 'Ford')",
   'field_name': "('bar', 'Ford')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None},
  {'name': "('bar', 'Alfa')",
   'field_name': "('bar', 'Alfa')",
   'pandas_type': 'int64',
   'numpy_type': 'int64',
   'metadata': None}],
 'creator': {'library': 'pyarrow', 'version': '8.0.0.dev589+g57f1a166b'},
 'pandas_version': '1.4.1'}
{code}



> [Python]  Multiindex levels order is not preserved after a from_pandas/to_pandas
> --------------------------------------------------------------------------------
>
>                 Key: ARROW-16580
>                 URL: https://issues.apache.org/jira/browse/ARROW-16580
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: Python
>    Affects Versions: 8.0.0
>            Reporter: Fabien Aulaire
>            Priority: Minor
>
> Hello,
> Not sure if it's the good place to report this issue but here is what I saw when I tried to convert a multi indexed dataframe (columns) to Table and convert it back to pandas:
>  
> {code:python}
> import pandas as pd
> import pyarrow
> # pandas version '1.4.1'
> # pyarrow version '8.0.0'
> df = pd.DataFrame([[100,300, 400], [200,500, 600]], columns=['Toyota', 'Ford', 'Alfa'])
> concatenated = pd.concat([df, df*2], axis=1, keys=['foo', 'bar'])
> concatenated.columns.names = ['l1', 'l2']
> table = pyarrow.Table.from_pandas(concatenated)
> from_table_df = table.to_pandas()
> from_table_df.columns.levels # == FrozenList([['bar', 'foo'], ['Alfa', 'Ford', 'Toyota']])
> concatenated.columns.levels # == FrozenList([['foo', 'bar'], ['Toyota', 'Ford', 'Alfa']])
> {code}
> the order of columns levels is not preserved. 
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)