You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2024/03/22 16:44:08 UTC
[I] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
0x26res opened a new issue, #40750:
URL: https://github.com/apache/arrow/issues/40750
### Describe the bug, including details regarding any error messages, version, and platform.
I'm trying to call `pa.MapArray.from_arrays`, in a simple example where I create a MapArray of empty maps:
```
offsets = pa.array([0, 0, 0, 0, 0, 0], pa.int32())
map_array = pa.MapArray.from_arrays(
offsets,
pa.array([], pa.string()),
pa.array([], pa.string()),
)
assert len(map_array) == 5
assert map_array.to_pylist() == [[]] * 5
```
It works fine but if I try the same thing with an offsetted view of `offsets` (aka a slice), it fails:
```
with pytest.raises(
pa.ArrowInvalid,
match=r"List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array \(0 < 1\)",
):
pa.MapArray.from_arrays(
offsets[1:],
pa.array([], pa.string()),
pa.array([], pa.string()),
)
```
For now as a work around I copy the data:
```
offset_shift = offsets[1:]
offset_shift_copy = pa.Array.from_buffers(
offset_shift.type, len(offset_shift), offset_shift.buffers()
)
map_array = pa.MapArray.from_arrays(
offset_shift_copy,
pa.array([], pa.string()),
pa.array([], pa.string()),
)
assert len(map_array) == 4
assert map_array.to_pylist() == [[]] * 4
```
My intuition is that the `offset` isn't interpreted correctly in the underlying code, but I can't really tell where.
For context, this is happening when I try to do my own cast of a chunked MapArray. I have to iterate through the chunks and change slightly the underlying value array, and this happens.
Tested with pyarrow==15.0.2 / Python 3.11.8
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #40750:
URL: https://github.com/apache/arrow/issues/40750#issuecomment-2017858344
Looking at the C++ code it seems the offset of an offsets array is applied to the child arrays https://github.com/apache/arrow/blob/2b0427559d94fb25ce61672c1ed17fc245d0a546/cpp/src/arrow/array/array_nested.cc#L796-L798
But in the case of empty child arrays this fails with
```
ArrowInvalid: List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array (0 < 1)
/Users/alenkafrim/repos/arrow/cpp/src/arrow/array/validate.cc:267 ValidateListLike(type)
```
and the source of the error can be found here:
https://github.com/apache/arrow/blob/2b0427559d94fb25ce61672c1ed17fc245d0a546/cpp/src/arrow/array/validate.cc#L309-L313
I think a check for this corner case where data is empty but the offset array has an offset can be added in `MapArray::FromArraysInternal`. In this case the offset should not be applied to the child arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #40750:
URL: https://github.com/apache/arrow/issues/40750#issuecomment-2018021152
> Looking at the C++ code it seems the offset of an offsets array is applied to the child arrays
>
> https://github.com/apache/arrow/blob/2b0427559d94fb25ce61672c1ed17fc245d0a546/cpp/src/arrow/array/array_nested.cc#L796-L798
>
> But in the case of empty child arrays this fails with
>
> ```
> ArrowInvalid: List child array invalid: Invalid: Struct child array #0 has length smaller than expected for struct array (0 < 1)
> /Users/alenkafrim/repos/arrow/cpp/src/arrow/array/validate.cc:267 ValidateListLike(type)
> ```
>
> and the source of the error can be found here:
>
> https://github.com/apache/arrow/blob/2b0427559d94fb25ce61672c1ed17fc245d0a546/cpp/src/arrow/array/validate.cc#L309-L313
>
> I think a check for this corner case where data is empty but the offset array has an offset can be added in `MapArray::FromArraysInternal`. In this case the offset should not be applied to the child arrays.
I might be confused, but shouldn't `field_data.length` be compared to the the offset of the last position (something like data.offsets[data.offset + data.length])?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF closed issue #40750: [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets
URL: https://github.com/apache/arrow/issues/40750
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #40750:
URL: https://github.com/apache/arrow/issues/40750#issuecomment-2020284706
> I might be confused, but shouldn't field_data.length be compared to the the offset of the last position (something like data.offsets[data.offset + data.length])?
Not sure.
I am thinking more in a direction that a check should be added here:
https://github.com/apache/arrow/blob/2b0427559d94fb25ce61672c1ed17fc245d0a546/cpp/src/arrow/array/validate.cc#L266-L269
for when the length of the children array is less than the length of offsets + the offset or something similar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #40750:
URL: https://github.com/apache/arrow/issues/40750#issuecomment-2100397277
Issue resolved by pull request 40871
https://github.com/apache/arrow/pull/40871
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Python] Getting "List child array invalid" when calling MapArray.from_arrays with offsetted offsets [arrow]
Posted by "AlenkaF (via GitHub)" <gi...@apache.org>.
AlenkaF commented on issue #40750:
URL: https://github.com/apache/arrow/issues/40750#issuecomment-2025455351
Opened a draft PR with an initial try to fix this issue after brainstorming the cause with Joris: https://github.com/apache/arrow/pull/40871.
Still plan to add one more tests with non-empty struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org