You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "alexeykudinkin (via GitHub)" <gi...@apache.org> on 2025/09/03 22:48:03 UTC

[I] Incorrect handling of both projection and filters pushed down [arrow]

alexeykudinkin opened a new issue, #47493:
URL: https://github.com/apache/arrow/issues/47493

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   We've recently discovered a following bug in the Arrow's Parquet reader:
   
    - When both filter and projection are projection is applied first, then filtering returning incorrect results
   
   ```
   import pyarrow as pa
   from pyarrow import compute as pac
   from pyarrow import parquet as pq
   
   path = '/tmp/test.parquet'
   
   t = pa.table({'ints': pa.array(list(range(100)))})
   
   pq.write_table(t, path)
   
   ds = pq.ParquetDataset(path)
   
   bs = ds.fragments[0].to_batches(
       columns=[],
       filter=pac.field('ints') == 42
   )
   
   assert len(list(bs)) == 1
   ```
   
   ### Component(s)
   
   C++, Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Incorrect handling of both projection and filters pushed down [arrow]

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on issue #47493:
URL: https://github.com/apache/arrow/issues/47493#issuecomment-3251049531

   Looking at the code it seems that the following is the culprit:
   
    - Specified `columns` aren't checked whether they include columns used in the filter
    - Batches are projected in the reader itself
    - However, filtering is performed by Acero based on read batches


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Incorrect handling of both projection and filters pushed down [arrow]

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin closed issue #47493: Incorrect handling of both projection and filters pushed down
URL: https://github.com/apache/arrow/issues/47493


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Incorrect handling of both projection and filters pushed down [arrow]

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on issue #47493:
URL: https://github.com/apache/arrow/issues/47493#issuecomment-3251853853

   Correcting myself on the record here -- asertion has to actually be 
   
   ```
   assert list(bs)[0].num_rows == 1
   ```
   
   And it actually works properly. Closing this


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org