You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Joris Van den Bossche (Jira)" <ji...@apache.org> on 2021/01/25 14:00:00 UTC
[jira] [Commented] (ARROW-11371) [Python] Serializing deeply nested
structures raises recursion error
[ https://issues.apache.org/jira/browse/ARROW-11371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17271325#comment-17271325 ]
Joris Van den Bossche commented on ARROW-11371:
-----------------------------------------------
Note that the pyarrow.serialize functionality is deprecated: see https://arrow.apache.org/docs/python/ipc.html#arbitrary-object-serialization
> [Python] Serializing deeply nested structures raises recursion error
> --------------------------------------------------------------------
>
> Key: ARROW-11371
> URL: https://issues.apache.org/jira/browse/ARROW-11371
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 1.0.1
> Environment: $ python --version
> Python 3.6.12
> $ pip list | grep pyarrow
> pyarrow 1.0.1
> Reporter: Alexander Dmitriev
> Priority: Minor
> Labels: pyarrow-serialization
> Attachments: pyarrow_recursion.py
>
>
> When serializing deeply nested python structures such as:
> {code:java}
> data = [[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[None]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]{code}
> orĀ
> {code:java}
> data = OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', OrderedDict([('a', None)]))]))]))]))]))]))]))]))]){code}
> pyarrow raises the following exception:
> {code:java}
> >>> buffer = pyarrow.serialize(data).to_buffer()
> Traceback (most recent call last):
> File "pyarrow_recursion.py", line 11, in try_recursion
> buffer = pyarrow.serialize(data).to_buffer()
> File "pyarrow/serialization.pxi", line 285, in pyarrow.lib.SerializedPyObject.to_buffer
> File "pyarrow/serialization.pxi", line 248, in pyarrow.lib.SerializedPyObject.total_bytes.__get__
> File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: Max recursion depth reached
> {code}
> While this not the prime use case of pyarrow.serialize, it would be nice to have a much larger recursion limit here, to not run into this issue with auto-generated structures quite so often.
> An example script for reproducing the error can be found in the attachments.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)