You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2026/03/26 14:49:35 UTC
Re: [I] Filtering corrupts data in column containing an array [arrow]
AlenkaF commented on issue #49392:
URL: https://github.com/apache/arrow/issues/49392#issuecomment-4135599337
Thank you for opening the issue! This looks like a serious bug to me.
From testing with PyArrow and investigating with Copilot, it looks like overflow happens in the "offset arithmetic".
Take for example a table with `large_list` instead of `list` column:
```python
>>> my_schema = pa.schema([
... pa.field('id', pa.int64()),
... pa.field('text', pa.string()),
... pa.field('numbers', pa.large_list(pa.float64()))])
>>> tbl = pa.table([ids, texts, numbers], schema=my_schema)
>>> tbl.filter(pc.field("id") == N - 1)
pyarrow.Table
id: int64
text: string
numbers: large_list<item: double>
child 0, item: double
----
id: [[],[],...,[],[499999]]
text: [[],[],...,[],["Row 499999 with data"]]
numbers: [[],[],...,[],[[499999,0.2806802660498191,0.18948458094650322,0.6611584406407851,0.340530752637791,...,0.19918275933231844,0.42906946186903017,0.49644347191463034,0.3171420306034032,0.13584405454197468]]]
```
so this bug would need a fix in the C++ filter kernel.
For now, I think the schema definition or casting list column to large list can be used as a workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org