Re: [I] [Python] Accessing parquet files with parquet.read_table in google cloud storage fails, but works with dataset, works in 16.1.0 fails in 17.0.0 [arrow]

["bkurtz (via GitHub)" <gi...@apache.org>]

bkurtz commented on issue #43574:
URL: https://github.com/apache/arrow/issues/43574#issuecomment-2447639750

   I'll chime in here, since I believe this is a behavior change that was unintentionally introduced in pyarrow 17.0 via #39438, specifically the two lines under discussion here: https://github.com/apache/arrow/pull/39438/files#r1469251517
   
   Previously, specifying a single file would always just return the contents of the file; now it will include any hive partition metadata columns that happen to be detected in the path.
   
   In particular, this is frustrating because it behaves differently for files on the local filesystem (the file is a `BufferedReader` rather than a path at that point, so hive metadata in the leading portion of the path is not interpreted) vs on cloud storage like AWS or S3.
   
   Other bits that made this unexpected and/or could be improved if it's considered desirable to keep this change:
   * the behavior change was not noted in the release notes (as it appears to be unexpected); had to consult the source code to find the offending change
   * `pandas` uses `read_table` as its entrypoint into pyarrow
   * the docs for `read_table` don't make it particularly clear that calling with `partitioning=None` is an option


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org