[I] Pyarrow and Parquet reading : int32 (Date) is treated as object [arrow]

["simonaubertbd (via GitHub)" <gi...@apache.org>]

simonaubertbd opened a new issue, #47464:
URL: https://github.com/apache/arrow/issues/47464

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   Hello,
   
   When I have a date in my parquet file (exactly an int32 with DATE annotation), it's interpretated as an object.
   As a reminder : https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
   I think it's linked to #47460  Maybe it's the annotation for logical types that are wrongly tested?
   
   Best regards,
   
   Simon
   
   ### Component(s)
   
   Python, Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python][Parquet] When reading, int32 (Date) is treated as object [arrow]

["jorisvandenbossche (via GitHub)" <gi...@apache.org>]

jorisvandenbossche closed issue #47464: [Python][Parquet] When reading, int32 (Date) is treated as object
URL: https://github.com/apache/arrow/issues/47464


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Python][Parquet] When reading, int32 (Date) is treated as object [arrow]

["jorisvandenbossche (via GitHub)" <gi...@apache.org>]

jorisvandenbossche commented on issue #47464:
URL: https://github.com/apache/arrow/issues/47464#issuecomment-3257645492

   Similarly as in https://github.com/apache/arrow/issues/47460, the issue is the conversion from Arrow->pandas, and not in reading the parquet file.
   
   Parquet has a date type, and Arrow as well. And so when reading the Parquet file into an Arrow table using pyarrow, you would see that the type is preserved. You can try:
   
   ```python
   import pyarrow.parquet as pq
   table = pq.read_table("C:/Users/qnsv2207/Desktop/test_amphi_compare.parquet")
   table
   ```
   
   This should show that the "OrderDate" column has a `date32` type.
   
   But then pandas does not have a built-in "date" type. Therefore, in the arrow->pandas conversion, pyarrow by default converts its date types into an `object` column with python datetime.date objects. 
   
   See the documentation about this at https://arrow.apache.org/docs/python/pandas.html#date-types, which also mentions the `date_as_object=False` option you can specify in `to_pandas()` to avoid this conversion to object dtype.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Parquet] When reading, int32 (Date) is treated as object [arrow]

["pitrou (via GitHub)" <gi...@apache.org>]

pitrou commented on issue #47464:
URL: https://github.com/apache/arrow/issues/47464#issuecomment-3254034765

   What if you don't call `to_pandas`?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org