[jira] [Created] (ARROW-13413) IPC roundtrip fails in to_pandas with empty table and extension type

["Thomas Buhrmann (Jira)" <ji...@apache.org>]

Thomas Buhrmann created ARROW-13413:
---------------------------------------

             Summary: IPC roundtrip fails in to_pandas with empty table and extension type 
                 Key: ARROW-13413
                 URL: https://issues.apache.org/jira/browse/ARROW-13413
             Project: Apache Arrow
          Issue Type: Bug
          Components: C++, Python
    Affects Versions: 4.0.1
            Reporter: Thomas Buhrmann


With pyarrow=4.0.1 and pandas=1.2.3, when writing then reading an empty DataFrame with an extension dtype, `to_pandas` subsequently fails to convert the arrow table:
{code:python}
import pandas as pd
import pyarrow as pa

df1 = pd.DataFrame({"x": pd.Series([], dtype="Int8")})
tbl1 = pa.Table.from_pandas(df1)

# In memory roundtrip seems to work fine
pa.Table.from_pandas(tbl1.to_pandas()).to_pandas()

path = "/tmp/tmp.arr"
writer = pa.RecordBatchStreamWriter(path, tbl1.schema)
writer.write_table(tbl1)
writer.close()
reader = pa.RecordBatchStreamReader(path)
tbl2 = reader.read_all()

assert tbl1.schema.equals(tbl2.schema)
assert tbl2.schema.metadata == tbl2.schema.metadata

df2 = tbl1.to_pandas()
try:
    df2 = tbl2.to_pandas()
except Exception as e:
    print(f"Error: {e}")
    df2 = tbl2.replace_schema_metadata(None).to_pandas()
{code}
In the above example (with `Int8` as the pandas dtype), the table read from disk cannot be converted to a DataFrame, even though its schema and metadata are supposedly equal  to the original table. Removing its metadata "fixes" the issue.

The problem doesn't occur with "normal" dtypes. This may well be a bug in Pandas, but it seems to depend on some change in Arrow's metadata.

The full stacktrace:
{code:java}
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-08855adb276d> in <module>
----> 1 df2 = tbl2.to_pandas()

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib._PandasConvertible.to_pandas()

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.Table._to_pandas()

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/pandas_compat.py in table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)
    787     _check_data_column_metadata_consistency(all_columns)
    788     columns = _deserialize_column_index(table, all_columns, column_indexes)
--> 789     blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
    790 
    791     axes = [columns, index]

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/pandas_compat.py in _table_to_blocks(options, block_table, categories, extension_columns)
   1128     result = pa.lib.table_to_blocks(options, block_table, categories,
   1129                                     list(extension_columns.keys()))
-> 1130     return [_reconstruct_block(item, columns, extension_columns)
   1131             for item in result]
   1132 

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/pandas_compat.py in <listcomp>(.0)
   1128     result = pa.lib.table_to_blocks(options, block_table, categories,
   1129                                     list(extension_columns.keys()))
-> 1130     return [_reconstruct_block(item, columns, extension_columns)
   1131             for item in result]
   1132 

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pyarrow/pandas_compat.py in _reconstruct_block(item, columns, extension_columns)
    747             raise ValueError("This column does not support to be converted "
    748                              "to a pandas ExtensionArray")
--> 749         pd_ext_arr = pandas_dtype.__from_arrow__(arr)
    750         block = _int.make_block(pd_ext_arr, placement=placement)
    751     else:

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pandas/core/arrays/integer.py in __from_arrow__(self, array)
    119             results.append(int_arr)
    120 
--> 121         return IntegerArray._concat_same_type(results)
    122 
    123 

~/miniforge3/envs/grapy/lib/python3.8/site-packages/pandas/core/arrays/masked.py in _concat_same_type(cls, to_concat)
    269         cls: Type[BaseMaskedArrayT], to_concat: Sequence[BaseMaskedArrayT]
    270     ) -> BaseMaskedArrayT:
--> 271         data = np.concatenate([x._data for x in to_concat])
    272         mask = np.concatenate([x._mask for x in to_concat])
    273         return cls(data, mask)

<__array_function__ internals> in concatenate(*args, **kwargs)

ValueError: need at least one array to concatenate
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)