[GitHub] [arrow] jorisvandenbossche commented on issue #37355: [Python] pyarrow.parquet.read_table with filters is broken for timezone aware datetime since 13.0.0 release

["jorisvandenbossche (via GitHub)" <gi...@apache.org>]

jorisvandenbossche commented on issue #37355:
URL: https://github.com/apache/arrow/issues/37355#issuecomment-1692915164

   A simpler reproducer:
   
   ```python
   import pyarrow as pa
   import pyarrow.compute as pc
   
   # create table with tz-aware nanosecond resolution timestamp
   table = pa.table({'timestamp': pa.array([1], pa.timestamp("ns", "UTC"))})
   
   # comparison with microseconds works if there are microseconds
   table.filter(pc.field("timestamp") <= pa.scalar(1, pa.timestamp("us", "UTC")))
   
   # comparison fails with microseconds if there are no microseconds 
   table.filter(pc.field("timestamp") <= pa.scalar(0, pa.timestamp("us", "UTC")))
   # ...
   # ArrowNotImplementedError: Function 'less_equal' has no kernel
   # matching input types (timestamp[ns, tz=UTC], timestamp[s])
   
   # but works again if the resolution matches
   table.filter(pc.field("timestamp") <= pa.scalar(0, pa.timestamp("ns", "UTC")))
   ```
   
   It somehow completely looses the type information of the scalar (both the resolution and the timezone) somewhere inside Acero.
   
   Calling the compute kernel directly instead of going through an expression and execute with Acero seems to work fine:
   
   ```
   >>> pc.less_equal(table["timestamp"], pa.scalar(0, pa.timestamp("us", "UTC")))
   <pyarrow.lib.ChunkedArray object at 0x7f804bbb0c20>
   [
     [
       false
     ]
   ]
   ```
   
   ---
   
   The above reproducer actually also fails with pyarrow 12.0.0, but still seems fixed on the main branch. So some other change in 13.0.0 might have changed the dataset filtering to take the code path from the reproducer above.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org