You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org> on 2026/07/28 14:29:03 UTC
[PR] [python] Prune Parquet row groups for row-id range reads [paimon]
XiaoHongbo-Hope opened a new pull request, #8893:
URL: https://github.com/apache/paimon/pull/8893
### Purpose
Data-evolution scans can already carry exact global row-id ranges in an
`IndexedSplit`. Native BLOB, Vortex, Lance, and ROW readers consume those
ranges directly, but the PyArrow Parquet path reads every row in each
intersecting file and applies `RowIdFilterRecordBatchReader` afterwards.
That fallback causes severe read amplification after compaction creates a
large Parquet file: a sparse indexed lookup still decodes the whole file and
executes a Python range check for every row.
This change adds native Parquet row-range reads at row-group granularity:
- translate global row-id ranges to file-local row indices;
- open only the Parquet row groups intersecting those indices;
- slice emitted batches to the exact requested rows while preserving order;
- pass the selected indices to `DataFileBatchReader` so generated `_ROW_ID`
values stay aligned;
- skip `RowIdFilterRecordBatchReader` on this native path.
Correctness guard: the optimization is enabled only when there is no
scanner-level residual predicate. A scanner predicate removes rows before
position-based slicing and could shift positions, so that case deliberately
keeps the existing whole-file read plus row-id filter. Full scans and other
formats are unchanged. Projected VARIANT columns use the same selected row
groups and exact slicer.
### Read amplification reproduction
Read-only reproduction against a compacted data-evolution BLOB table, using an
indexed equality filter that selected 713 descriptor rows from one Parquet
file containing 24,836,412 rows:
| Metric | Before | After |
| --- | ---: | ---: |
| Returned rows | 713 | 713 |
| Descriptor bytes | 181,815 | 181,815 |
| Per-row Python range checks | 24,836,412 | 0 |
| Descriptor read time (warm environment) | 13.390 s | 0.876 s |
The ordered result SHA-256 was identical before and after.
### Tests
```text
python -m unittest \
pypaimon.tests.parquet_row_range_test \
pypaimon.tests.format_pyarrow_variant_row_group_test \
pypaimon.tests.projection_predicate_index_test \
pypaimon.tests.data_evolution_test
Ran 59 tests in 4.797s
OK
```
New tests cover intersecting-row-group selection, disjoint ranges, batch and
row-group boundaries, payload alignment, bypassing the per-row Python filter,
residual column-predicate fallback, projected VARIANT reads, and unchanged
full scans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Prune Parquet row groups for row-id range reads [paimon]
Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on code in PR #8893:
URL: https://github.com/apache/paimon/pull/8893#discussion_r3667292744
##########
paimon-python/pypaimon/read/reader/format_pyarrow_reader.py:
##########
@@ -115,12 +167,15 @@ def __init__(self, file_io: FileIO, file_format: str, file_path: str,
# Read projected VARIANT columns in bounded batches.
self._parquet_file = None
- if self._bounded_variant_read:
+ if (self._bounded_variant_read
+ or self._selected_parquet_row_groups is not None):
Review Comment:
Fixed in d29233b98. The row-group path now preserves the logical row count when rebuilding a zero-column RecordBatch. Added regressions for both `_ROW_ID`-only and missing-only projections; the related suite passes on the current environment and on Python 3.6 with PyArrow 6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Prune Parquet row groups for row-id range reads [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8893:
URL: https://github.com/apache/paimon/pull/8893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Prune Parquet row groups for row-id range reads [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #8893:
URL: https://github.com/apache/paimon/pull/8893#discussion_r3667219876
##########
paimon-python/pypaimon/read/reader/format_pyarrow_reader.py:
##########
@@ -115,12 +167,15 @@ def __init__(self, file_io: FileIO, file_format: str, file_path: str,
# Read projected VARIANT columns in bounded batches.
self._parquet_file = None
- if self._bounded_variant_read:
+ if (self._bounded_variant_read
+ or self._selected_parquet_row_groups is not None):
Review Comment:
Blocking: this new row-group path also handles projections with no physical Parquet columns. In that case, `_select_existing_fields(batch)` rebuilds the zero-column batch with `pa.RecordBatch.from_arrays([], ...)`, resetting its row count to 0. This occurs for `with_projection(["_ROW_ID"])` and for schema-evolution files where every projected column is missing. With `_ROW_ID BETWEEN 2 AND 7`, the base commit returns `[2, 3, 4, 5, 6, 7]`, but this branch returns no rows on PyArrow 16.0.0. Please preserve the row count of zero-column batches and add a system-only/missing-only projection regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org