You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Krisztian Szucs (Jira)" <ji...@apache.org> on 2020/09/14 15:21:00 UTC
[jira] [Created] (ARROW-9994) [C++][Python] Auto chunking nested
array containing binary-like fields result malformed output
Krisztian Szucs created ARROW-9994:
--------------------------------------
Summary: [C++][Python] Auto chunking nested array containing binary-like fields result malformed output
Key: ARROW-9994
URL: https://issues.apache.org/jira/browse/ARROW-9994
Project: Apache Arrow
Issue Type: Improvement
Components: C++, Python
Affects Versions: 1.0.0
Reporter: Krisztian Szucs
Assignee: Krisztian Szucs
In case of nested types the binary-like arrays are chunked but not the others, so after finalizing the builder the nested output array contains different length children.
{code:python}
char = b'x'
ty = pa.binary()
v1 = char * 100000000
v2 = char * 147483646
struct_type = pa.struct([
pa.field('bool', pa.bool_()),
pa.field('integer', pa.int64()),
pa.field('string-like', ty),
])
data = [{'bool': True, 'integer': 1, 'string-like': v1}] * 20
data.append({'bool': True, 'integer': 1, 'string-like': v2})
arr = pa.array(data, type=struct_type)
assert isinstance(arr, pa.Array)
data.append({'bool': True, 'integer': 1, 'string-like': char})
arr = pa.array(data, type=struct_type)
assert isinstance(arr, pa.ChunkedArray)
{code}
{code:python}
len(arr.field(0)) == 22
len(arr.field(1)) == 22
len(arr.field(2)) == 1 # the string array gets chunked whereas the rest of the fields do not
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)