[jira] [Updated] (ARROW-9994) [C++][Python] Auto chunking nested array containing binary-like fields result malformed output

["Krisztian Szucs (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-9994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Krisztian Szucs updated ARROW-9994:
-----------------------------------
    Issue Type: Bug  (was: Improvement)

> [C++][Python] Auto chunking nested array containing binary-like fields result malformed output
> ----------------------------------------------------------------------------------------------
>
>                 Key: ARROW-9994
>                 URL: https://issues.apache.org/jira/browse/ARROW-9994
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Python
>    Affects Versions: 1.0.0
>            Reporter: Krisztian Szucs
>            Assignee: Krisztian Szucs
>            Priority: Major
>
> In case of nested types the binary-like arrays are chunked but not the others, so after finalizing the builder the nested output array contains different length children.
> {code:python}
>    char = b'x'
>    ty = pa.binary()
>     v1 = char * 100000000
>     v2 = char * 147483646
>     struct_type = pa.struct([
>         pa.field('bool', pa.bool_()),
>         pa.field('integer', pa.int64()),
>         pa.field('string-like', ty),
>     ])
>     data = [{'bool': True, 'integer': 1, 'string-like': v1}] * 20
>     data.append({'bool': True, 'integer': 1, 'string-like': v2})
>     arr = pa.array(data, type=struct_type)
>     assert isinstance(arr, pa.Array)
>     data.append({'bool': True, 'integer': 1, 'string-like': char})
>     arr = pa.array(data, type=struct_type)
>     assert isinstance(arr, pa.ChunkedArray)
> {code}
> {code:python}
> len(arr.field(0)) == 22
> len(arr.field(1)) == 22
> len(arr.field(2)) == 1  # the string array gets chunked whereas the rest of the fields do not
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)