[I] Sequential reading of parquet files [arrow]

["alippai (via GitHub)" <gi...@apache.org>]

alippai opened a new issue, #45298:
URL: https://github.com/apache/arrow/issues/45298

   ### Describe the enhancement requested
   
   I'm using Parquet via PyArrow (versions 17.0 and 18.0).
   
   I have tried many options (setting `pre_buffer=True`, `use_threads=False`, increasing `buffer_size`) with `pq.read_table()`, but I can't find a way to reduce seeking and increase the size of `read()` calls.
   
   As I understand, the Parquet format can be read by reading the footer once or twice, then streaming the row groups in order and passing the data to decompress and decode.
   
   If I have a reasonable number of files, having a few sequential readers (one reader per file) should provide the most throughput when the storage is inefficient with small random reads.
   
   Currently, I use `pq.read_table(BytesIO(Path().read_bytes()))`. This is wasteful and clunky: it allocates too much contiguous memory, and the read doesn't free the processed row groups.
   
   I understand this can get complex when filters and column projection are involved, but developing an I/O plan where I can specify in-order reads and a minimum read size could work (e.g., reading the small gaps and discarding unused data afterward).
   
   ### Component(s)
   
   Parquet, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [C++][Parquet][Python] Sequential reading of parquet files [arrow]

["github-actions[bot] (via GitHub)" <gi...@apache.org>]

github-actions[bot] commented on issue #45298:
URL: https://github.com/apache/arrow/issues/45298#issuecomment-4430186566

   This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org