You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "Alexander Dmitriev (Jira)" <ji...@apache.org> on 2021/01/25 10:43:00 UTC
[jira] [Created] (ARROW-11371) [Python] Serializing deeply nested
structures raises recursion error
Alexander Dmitriev created ARROW-11371:
------------------------------------------
Summary: [Python] Serializing deeply nested structures raises recursion error
Key: ARROW-11371
URL: https://issues.apache.org/jira/browse/ARROW-11371
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 1.0.1
Environment: $ python --version
Python 3.6.12
$ pip list | grep pyarrow
pyarrow 1.0.1
Reporter: Alexander Dmitriev
Attachments: pyarrow_recursion.py
When serializing deeply nested python structures such as:
{code:java}
data = [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[None]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]{code}
orĀ
{code:java}
data = OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', None)]))]))]))]))]))]))]))]))]){code}
pyarrow raises the following exception:
{code:java}
>>> buffer = pyarrow.serialize(data).to_buffer()
Traceback (most recent call last):
File "pyarrow_recursion.py", line 11, in try_recursion
buffer = pyarrow.serialize(data).to_buffer()
File "pyarrow/serialization.pxi", line 285, in pyarrow.lib.SerializedPyObject.to_buffer
File "pyarrow/serialization.pxi", line 248, in pyarrow.lib.SerializedPyObject.total_bytes.__get__
File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Max recursion depth reached
{code}
While this not the prime use case of pyarrow.serialize, it would be nice to have a much larger recursion limit here, to not run into this issue with auto-generated structures quite so often.
An example script for reproducing the error can be found in the attachments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)