Re: [I] [R] [parquet] Cannot load chunked large_list from parquet file [arrow]

["thisisnic (via GitHub)" <gi...@apache.org>]

thisisnic commented on issue #45087:
URL: https://github.com/apache/arrow/issues/45087#issuecomment-5985250877

   Thanks for the really clear reprex, it made this easy to track down!
   I had a go at reproducing this on main and it turns out to be the same underlying issue as #32723 rather than anything R-specific. `pq.read_table()` only works because it goes through the dataset API, which reads one row group at a time; if you call `pq.ParquetFile("t.parquet").read()` in Python you get the exact same error as R does.
   `read_parquet()` in R calls `ReadTable()`, which reads all the row groups in one go, and the dictionary leaf comes back as one chunk per row group, which the list reader can't handle yet.
   In the meantime, `open_dataset("t.parquet") |> dplyr::collect()` should work fine for you, as it takes the same path as `pq.read_table()`.
   I've reopened #32723 as that's where the fix needs to happen, so I'm going to close this one as a duplicate, but do shout if the workaround doesn't do what you need.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org