[I] [Python][Parquet] Expose low-level page reader in pyarrow.parquet [arrow]

["JigaoLuo (via GitHub)" <gi...@apache.org>]

JigaoLuo opened a new issue, #53993:
URL: https://github.com/apache/arrow/issues/53993

   ### Describe the enhancement requested
   
   Parquet C++ already provides page-level access via `RowGroupReader::GetColumnPageReader()` and `PageReader::NextPage()` (see `cpp/src/parquet/column_page.h`), but PyArrow only exposes the Arrow-level reader (`read_table`, `read_row_group`, `iter_batches`). From Python, the smallest unit a Parquet file can be read at is a row group.
   
   #### Motivation: reducing read amplification
   
   A row group is typically tens to hundreds of MB and contains many pages per column. When a Python user only needs a small part of a column chunk (a point lookup, a narrow row range, one page's statistics), PyArrow still has to fetch and decode the entire row group. On object storage this turns a sub-millisecond, few-KB page read into a multi-MB request. The page is the natural I/O unit of the format, and the C++ reader already works at that granularity; it is only the Python binding that hides it.
   
   #### Other use cases
   
   - Inspecting/debugging files at page level (page type, encoding, `num_values`, page statistics), similar to `parquet-cli pages`
   - Custom or experimental decoding on top of PyArrow's I/O and decompression
   - Page-level processing without full Arrow materialization. Downstream projects currently re-implement page parsing in Python to get this (e.g. apache/paimon#9850)
   
   #### Prior art
   
   The Lance file format's Python binding (pylance) works at page granularity. `LanceFileReader.metadata()` exposes each column's page layout (buffer positions/sizes and encoding), and the same page-level reader is what makes `LanceFileReader.take_rows()` / `LanceDataset.take()` efficient random access from Python without reading whole column chunks. Parquet C++ already has the equivalent reader; exposing it in PyArrow would give Python users the same building block.
   
   #### Rough API idea (open to suggestions)
   
   ```python
   pf = pq.ParquetFile("data.parquet")
   for page in pf.iter_pages(row_group=0, column=0):
       page.page_type   # DICTIONARY_PAGE / DATA_PAGE / DATA_PAGE_V2
       page.encoding
       page.num_values
       page.buffer      # pyarrow.Buffer, decompressed page body
   ```
   
   The C++ functionality exists and is stable; this would be a read-only wrapper in the Python binding layer, with no new C++ code required. I'm happy to work on a PR.
   
   ### Component(s)
   
   Python, Parquet


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org