You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "AlenkaF (via GitHub)" <gi...@apache.org> on 2026/03/26 14:49:35 UTC

Re: [I] Filtering corrupts data in column containing an array [arrow]

AlenkaF commented on issue #49392:
URL: https://github.com/apache/arrow/issues/49392#issuecomment-4135599337

   Thank you for opening the issue! This looks like a serious bug to me.
   From testing with PyArrow and investigating with Copilot, it looks like overflow happens in the "offset arithmetic". 
   
   Take for example a table with `large_list` instead of `list` column:
   
   ```python
   >>> my_schema = pa.schema([
   ...     pa.field('id', pa.int64()),
   ...     pa.field('text', pa.string()),
   ...     pa.field('numbers', pa.large_list(pa.float64()))])
   >>> tbl = pa.table([ids, texts, numbers], schema=my_schema)
   
   >>> tbl.filter(pc.field("id") == N - 1)
   pyarrow.Table
   id: int64
   text: string
   numbers: large_list<item: double>
     child 0, item: double
   ----
   id: [[],[],...,[],[499999]]
   text: [[],[],...,[],["Row 499999 with data"]]
   numbers: [[],[],...,[],[[499999,0.2806802660498191,0.18948458094650322,0.6611584406407851,0.340530752637791,...,0.19918275933231844,0.42906946186903017,0.49644347191463034,0.3171420306034032,0.13584405454197468]]]
   ```
   
   so this bug would need a fix in the C++ filter kernel.
   
   For now, I think the schema definition or casting list column to large list can be used as a workaround.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org