You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/09/27 17:06:22 UTC
[I] [EPIC] Batch-granular fetch and decode in ParquetPushDecoder [arrow-rs]
adriangb opened a new issue, #11234:
URL: https://github.com/apache/arrow-rs/issues/11234
Part of https://github.com/apache/arrow-rs/issues/6946.
# Goal
Let `ParquetPushDecoder` request and decode one batch at a time, so that a caller gets the first batch of a row group as soon as the pages of that batch are pushed, and the decoder holds approximately one batch of bytes, not the full row group.
```rust
let mut decoder = ParquetPushDecoderBuilder::try_new_decoder(metadata)?
.with_fetch_granularity(FetchGranularity::Batch) // new, opt-in
.build()?;
// NeedsData asks only for the pages of the next batch.
```
Prototype measurement (one row group of 84 MB, 50 ms request latency):
| Mode | First batch | Peak buffered bytes |
|---|---:|---:|
| `FetchGranularity::RowGroup` (default) | 451 ms | 84 MB |
| `FetchGranularity::Batch`, read-ahead of 8 MB | 89 ms | 8.1 MB |
https://github.com/apache/arrow-rs/pull/11223 has the full change (+4,156 lines). It is too large to review in one step, so this EPIC splits it.
# Plan
Each PR is useful without the PRs after it. No PR adds a temporary error or fallback that a later PR removes. A later PR can add an optimization to code that is already correct.
| PR | Content | Base | Useful alone because |
|---|---|---|---|
| **A** https://github.com/apache/arrow-rs/pull/11218 | Test: the decoder needs the full row group before it returns a batch | `main` | Records the current behavior |
| **C** TBD | `into_builder`: release the pushed bytes of row groups that the new decoder does not read | `main` | Today these bytes stay until `clear_all_ranges` |
| **D** TBD | Keep `PushBuffers` sorted by offset | `main` | Lookups become a binary search. Today each lookup scans all buffers. |
| **E** TBD | `FetchGranularity::Batch`, with and without a `RowFilter`. Holds the bytes of a row group until the row group ends. | A, #11233 fix | Lower time to the first batch |
| **F** TBD | Batch mode: release each data page when all readers have passed it | E, D | Lower peak memory |
| **G** TBD | Batch mode: use the predicate cache | E | Filtered scans do not decode predicate columns two times |
| **H** TBD | Batch mode: use the configured `RowSelectionPolicy` (mask) in each window | E | Faster dense selections |
```text
main ──┬── A test ─────────────────────┐
├── #11233 fix ──────────────────┴── E batch granularity ──┬── F page release ◀── also needs D
├── C into_builder release ├── G predicate cache
└── D sorted PushBuffers ──────────────────────────────────┴── H selection policy
```
A, C, D and the #11233 fix are independent and can be reviewed in parallel. F, G and H are independent of each other.
# Tangential
- https://github.com/apache/arrow-rs/issues/11233: the decoder does not release a pushed buffer that is larger than the requested ranges. This is a bug in the default mode. E uses the fix (`PushBuffers::release_ranges`), but the fix is not part of this EPIC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Batch-granular fetch and decode in ParquetPushDecoder [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11234:
URL: https://github.com/apache/arrow-rs/issues/11234#issuecomment-5859423668
(sorry for the ping above andrew, i told it to post a summary so i could read it and then I was going to ping you after vetting, i always try to not have an ai ping a human, appoligies. that said i've now read the comment above and it is reasonable 😄)
cc @sunchao
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Batch-granular fetch and decode in ParquetPushDecoder [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11234:
URL: https://github.com/apache/arrow-rs/issues/11234#issuecomment-6097375160
# PR stack for batch-granular decode
```text
main ──┬── 11484 fix ──────┐
└── 11485 tests ────┴── 11238 batch mode ── 11487 skip empty decode ── 11240 batch-mode optimizations
```
| Order | PR | Content | Size of the PR alone | Depends on | State |
|---|---|---|---:|---|---|
| 1 | https://github.com/apache/arrow-rs/pull/11484 | Fix: an offset index page past its column chunk gives an error, not a panic | +68 | `main` | Ready for review |
| 1 | https://github.com/apache/arrow-rs/pull/11485 | Tests only: the push decoder against the sync reader, with fuzz tests | +709 | `main` | Ready for review |
| 2 | https://github.com/apache/arrow-rs/pull/11238 | `FetchGranularity::Batch`: the first batch of a row group as soon as its pages are pushed | [+2,354](https://github.com/pydantic/arrow-rs/compare/fe5a3f59d8...29553db415) (about 1,100 are tests) | 11484, 11485 | Draft |
| 3 | https://github.com/apache/arrow-rs/pull/11487 | Batch mode: no empty decode after each batch | [+70](https://github.com/pydantic/arrow-rs/compare/29553db415...5b6b92d38a) | 11238 | Draft |
| 4 | https://github.com/apache/arrow-rs/pull/11240 | Batch mode: predicate cache, page release, configured selection policy | [+690](https://github.com/pydantic/arrow-rs/compare/5b6b92d38a...0f5ce5b793) | 11487 | Draft |
The two PRs of order 1 do not depend on each other. Each draft PR contains the commits of the PRs that it depends on, because a PR against this repository must have `main` as its base. The size links show the diff of that PR alone.
Already merged: https://github.com/apache/arrow-rs/pull/11218, https://github.com/apache/arrow-rs/pull/11235, https://github.com/apache/arrow-rs/pull/11237.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Batch-granular fetch and decode in ParquetPushDecoder [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11234:
URL: https://github.com/apache/arrow-rs/issues/11234#issuecomment-5859399040
@alamb, as you asked on https://github.com/apache/arrow-rs/pull/11223#issuecomment-5857360169, I split https://github.com/apache/arrow-rs/pull/11223 (+4,156 lines) into the PRs below. This comment gives the reason for each cut and a suggested review order.
# How the split works
| Rule | Result |
|---|---|
| Each PR is useful without the PRs after it | The 4 PRs on `main` fix or speed up the current decoder. Each PR on top of the core adds one improvement to batch mode. |
| No PR adds a temporary error or fallback that a later PR removes | Predicates must be in the core PR, because a batch mode without predicates needs a fallback for filtered scans |
| Later PRs only add optimizations to code that is already correct | Page release, predicate cache and selection policy are follow-ups |
| One decode path | The core has one windowed engine. A scan without predicates is the case with zero predicates. This removed the separate no-predicate path of #11223 and about 600 lines. |
# Suggested review order
```text
Step 1 (in parallel, all on main) Step 2 Step 3 (in parallel)
#11218 test (yours) ─┐
#11235 bug fix #11233 ─┴──────▶ #11238 ──┬──▶ #11239 page release
#11236 rebuild releases bytes ├──▶ #11240 predicate cache
#11237 sorted PushBuffers └──▶ #11241 selection policy
```
| Step | PR | What users get | New lines |
|---|---|---|---:|
| 1 | https://github.com/apache/arrow-rs/pull/11218 | A test of the current behavior | +83 |
| 1 | https://github.com/apache/arrow-rs/pull/11235 | Fixes https://github.com/apache/arrow-rs/issues/11233: the decoder keeps all pushed bytes until the end of the scan if the caller merges the requested ranges | +311 |
| 1 | https://github.com/apache/arrow-rs/pull/11236 | `into_builder().build()` releases the pushed bytes of the row groups that the new decoder skips | +204 |
| 1 | https://github.com/apache/arrow-rs/pull/11237 | Sorted `PushBuffers`: `Nbuf/100000ranges` goes from 574 ms to 35 ms | +225 / −41 |
| 2 | https://github.com/apache/arrow-rs/pull/11238 | **`FetchGranularity::Batch`**: the first batch of a row group as soon as its pages are pushed | +2,847 (about 1,580 tests) |
| 3 | https://github.com/apache/arrow-rs/pull/11239 | Batch mode: peak memory of about one batch, not one row group | +171 |
| 3 | https://github.com/apache/arrow-rs/pull/11240 | Batch mode: filtered scans decode the predicate columns one time | +149 |
| 3 | https://github.com/apache/arrow-rs/pull/11241 | Batch mode: the configured `RowSelectionPolicy` in each window | +176 |
## Step 2: how to read #11238
It is the largest PR, because the core cannot be smaller without a temporary fallback. About half of it is tests. Suggested reading order:
1. The docs of `FetchGranularity` in `push_decoder/mod.rs`: the behavior that users see, and the I/O contract.
2. The docs of `IncrementalRowGroup` and `Stage` in `reader_builder/incremental.rs`: the design.
3. `IncrementalRowGroup::step` and `ColumnChunkPages`: the engine.
4. The wiring in `reader_builder/mod.rs`, `remaining.rs` and `push_decoder/mod.rs`, which is small.
5. `incremental_tests.rs`: most tests compare batch mode with the default mode on the same scan.
The PR description also records two decisions from review: windows stay `batch_size` rows of the row group, and the caller decides how far to read ahead.
# Merge notes
| PRs | Note |
|---|---|
| #11235, #11236, #11237 | All edit `util/push_buffers.rs`. The PR that merges last needs a small rebase: remove the duplicate `merge_ranges` and test helper, and update the maximum buffer length of #11237 in `release_ranges` and `retain_ranges`. |
| #11239, #11240 | If both merge, the second one adds one rule: a column in the predicate cache is released at cache-batch boundaries |
| #11238 and step 3 | I will rebase them after each merge |
https://github.com/apache/arrow-rs/pull/11223 stays open as a reference for the full change. I will close it when the stack merges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Batch-granular fetch and decode in ParquetPushDecoder [arrow-rs]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #11234:
URL: https://github.com/apache/arrow-rs/issues/11234#issuecomment-5936485117
I simplified the stack to make review easier. It is now 4 PRs instead of 7, and the issue description has the new plan:
- **Ready now, in parallel:** https://github.com/apache/arrow-rs/pull/11235 (now also contains https://github.com/apache/arrow-rs/pull/11236) and https://github.com/apache/arrow-rs/pull/11237 (benchmark bot: `Nbuf/100000ranges` 15× faster).
- **Draft until 11235 merges:** https://github.com/apache/arrow-rs/pull/11238.
- **Draft until 11238 merges:** https://github.com/apache/arrow-rs/pull/11240 (now also contains https://github.com/apache/arrow-rs/pull/11239 and https://github.com/apache/arrow-rs/pull/11241). The page release and the predicate cache must agree on when cached pages are released, so they are one PR.
The merge notes in my earlier comment no longer apply. The only remaining one: whichever of 11235 and 11237 merges second gets a small rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org