[jira] [Commented] (ARROW-16231) [C++][Python] IPC failure for dictionary with extension type with struct storage type

["Joris Van den Bossche (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-16231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17524230#comment-17524230 ] 

Joris Van den Bossche commented on ARROW-16231:
-----------------------------------------------

If I try to recreate this with a pure-pyarrow example, I get a different error:

 

{code}
import pyarrow as pa
from pyarrow.tests.test_extension_type import MyStructType

struct_array = pa.StructArray.from_arrays(
    [pa.array([0, 1], type="int64"), pa.array([1, 2], type="int64")],
    names=["left", "right"])
mystruct_array = pa.ExtensionArray.from_storage(MyStructType(), struct_array)
dict_array = pa.DictionaryArray.from_arrays(pa.array([0, 1, 0]), mystruct_array)

# roundtrip through Feather
from pyarrow import feather
feather.write_feather(pa.table({'a': dict_array}), "test_dict_ext_nested.feather")
feather.read_table("test_dict_ext_nested.feather")
{code}

gives

{code}
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-df8b416670f4> in <module>
----> 1 feather.read_table("test_dict_ext_nested.feather")

~/scipy/repos/arrow/python/pyarrow/feather.py in read_table(source, columns, memory_map, use_threads)
    242     table : pyarrow.Table
    243     """
--> 244     reader = _feather.FeatherReader(
    245         source, use_memory_map=memory_map, use_threads=use_threads)
    246 

~/scipy/repos/arrow/python/pyarrow/_feather.pyx in pyarrow._feather.FeatherReader.__cinit__()
~/scipy/repos/arrow/python/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/scipy/repos/arrow/python/pyarrow/types.pxi in pyarrow.lib.PyExtensionType.__arrow_ext_deserialize__()
TypeError: Expected storage type struct<left: int64, right: int64> but got dictionary<values=struct<left: int64, right: int64>, indices=int64, ordered=0>
{code}


> [C++][Python] IPC failure for dictionary with extension type with struct storage type
> -------------------------------------------------------------------------------------
>
>                 Key: ARROW-16231
>                 URL: https://issues.apache.org/jira/browse/ARROW-16231
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python
>            Reporter: Joris Van den Bossche
>            Priority: Major
>
> Report from [https://github.com/apache/arrow/issues/12899]
> Roundtripping through IPC/Feather using a dictionary type where the dictionary is an extension type with a nested storage type fails. Writing seems to work (but no idea if the written file is "correct", as trying to read the schema gives an error), but reading it back fails with {_}"ArrowInvalid: Ran out of field metadata, likely malformed"{_}.
> The original use case was from a pandas extension type (the pandas interval dtype is mapped to an arrow extension type with a struct type as storage, and in this case this interval type was further wrapped in a categorical (dictionary) type). A pandas-based test that reproduces this (can be added like this in {{{}test_feather.py{}}}):
> {code:python}
> @pytest.mark.pandas
> def test_dictionary_interval():
>     df = pd.DataFrame({'a': pd.cut(range(1, 10, 3), [-1, 5, 10])})
>     _check_pandas_roundtrip(df, version=2)
> {code}
> this gives:
> {code:java}
> $ pytest python/pyarrow/tests/test_feather.py::test_dictionary_interval
> ....
> ========================= FAILURES =================
> ____________ test_dictionary_interval _______________
> pyarrow/_feather.pyx:88: in pyarrow._feather.FeatherReader.read
> E   pyarrow.lib.ArrowInvalid: Ran out of field metadata, likely malformed
> E   ../src/arrow/ipc/reader.cc:266  GetFieldMetadata(field_index_++, out_)
> E   ../src/arrow/ipc/reader.cc:283  LoadCommon(type_id)
> E   ../src/arrow/ipc/reader.cc:324  Load(child_fields[i].get(), parent->child_data[i].get())
> E   ../src/arrow/ipc/reader.cc:529  loader.Load(&field, column.get())
> E   ../src/arrow/ipc/reader.cc:1188  ReadRecordBatchInternal( *message->metadata(), schema_, field_inclusion_mask_, context, reader.get())
> E   ../src/arrow/ipc/feather.cc:730  reader->ReadRecordBatch(i)
> pyarrow/error.pxi:100: ArrowInvalid
> {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)