You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@arrow.apache.org by "joechenrh (via GitHub)" <gi...@apache.org> on 2026/06/26 05:12:37 UTC

[I] feat(parquet): support streaming reads for large data pages [arrow-go]

joechenrh opened a new issue, #865:
URL: https://github.com/apache/arrow-go/issues/865

### Describe the enhancement requested

We have a valid Parquet file with a very large data page. The page itself is not malformed or illegal; it is just large enough that materializing the whole uncompressed page body is expensive.

One observed page has metadata similar to:

```text
type: DATA_PAGE
compressedSize: 1079936746
uncompressedSize: 1426870008
numValues: 233
encoding: PLAIN
definitionLevelEncoding: RLE
repetitionLevelEncoding: BIT_PACKED
hasCrc: true
```

The column is a string-like BYTE_ARRAY column. The page contains a small number of very large values, so the uncompressed page body is large even though `numValues` is small.

Today the Parquet reader materializes the full uncompressed page buffer before decoding values. Even with buffered stream reading enabled for the column chunk, this still requires allocating memory proportional to the page's uncompressed size. For the example above, that means allocating around 1.4 GiB for a single page before any batch-level output memory is considered.

It would be useful for the reader to support a streaming/lazy data page read path for large pages. For Data Page V1, the reader could parse repetition and definition levels first into small buffers, then leave the decompressed stream positioned at the encoded values and decode PLAIN BYTE_ARRAY values on demand. This avoids allocating the entire uncompressed page body while preserving the existing page layout semantics.

A possible first step would be an opt-in reader property or strategy that enables streaming value decoding for supported page shapes, for example:

- Data Page V1
- PLAIN encoding
- BYTE_ARRAY / string-like columns
- RLE definition levels
- RLE or BIT_PACKED repetition levels
- codecs that can be exposed as an `io.Reader` stream

Unsupported encodings/codecs/page layouts could continue to fall back to the existing materialized page path.

The goal is not to reject or special-case large pages as invalid, but to allow applications to read valid Parquet files with very large pages while keeping peak reader memory closer to the requested batch size plus small level buffers.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade commented on issue #865:
URL: https://github.com/apache/arrow-go/issues/865#issuecomment-5501425072

   This was implemented in #880 (`53a747fe`, "feat(parquet): opt-in streaming reads for large data pages") — the PR didn't use a closing keyword, so the issue stayed open.
   
   `ReaderProperties` now has an opt-in `PageStreamingEnabled` flag that decodes eligible data pages incrementally instead of materializing the full uncompressed page buffer. Eligibility closely matches what was proposed here: V1/V2 PLAIN-encoded data pages larger than 1 MiB, of a streaming-capable unencrypted codec (UNCOMPRESSED/GZIP/BROTLI/ZSTD), for supported physical types. Every other page is read whole as before, and decoded values are unchanged either way.
   
   Closing as resolved — please open a new issue if you hit a page shape that still isn't covered.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade commented on issue #865:
URL: https://github.com/apache/arrow-go/issues/865#issuecomment-4834581302

   Sounds good! Looking forward to reviewing it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade commented on issue #865:
URL: https://github.com/apache/arrow-go/issues/865#issuecomment-4810611724

   This is an interesting idea! I like it in theory, it will just be tricky to implement properly. Are you interested in giving implementing this a shot yourself?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "joechenrh (via GitHub)" <gi...@apache.org>.

joechenrh commented on issue #865:
URL: https://github.com/apache/arrow-go/issues/865#issuecomment-4842211021

   A sketch of the approach — would appreciate a sanity check on the shape before I build it out.
   
   **Goal:** opt-in streaming decode for large data pages so peak memory tracks the requested batch size rather than the uncompressed page size. Off by default, no behavior change for existing readers.
   
   **Scope (first cut):** Data Page V1 + V2, `PLAIN` encoding, types `int32/int64/float/double/FLBA/byte_array`. Codecs: `UNCOMPRESSED/GZIP/BROTLI/ZSTD` only — kept as an explicit allowlist rather than a [`StreamingCodec`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/compress/compress.go#L98) assertion, since Snappy's `NewReader` is the framed format while Parquet stores raw blocks, and LZ4_RAW has no streaming reader. Everything else falls back to the current materialized path.
   
   **Core idea — a `ValueBuffer` source between page and decoder.** Today decoders read from an in-memory `[]byte` ([`SetData`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/internal/encoding/types.go#L37)) sliced out of the fully-decompressed page ([`decompress`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/file/page_reader.go#L501)). Introduce:
   
   ```go
   type ValueBuffer interface {
       Bytes() []byte                           // currently buffered; valid until next Fill
       Fill(consumed, need int) ([]byte, error) // drop consumed, ensure >= need contiguous (grows for oversized values)
       Stable() bool                            // true => caller may zero-copy alias (buffer never slides)
   }
   ```
   
   Two implementations: a materialized one wrapping the existing page buffer (never refills, `Stable()=true`) and a streaming one wrapping `codec.NewReader(io.LimitReader(...))`. PLAIN decoders are reworked to index `Bytes()` directly and `Fill` only on underflow, so the materialized path is the degenerate "never refill" case and existing behavior is preserved (to be confirmed by benchmarks).
   
   **Pages — no new type.** [`DataPageV1`/`DataPageV2`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/file/page_reader.go#L135) gain a `valueSource` field (non-nil ⇒ streaming), and the [`DataPage`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/file/page_reader.go#L107) interface gains:
   
   ```go
   LevelData() []byte                               // bytes for the level decoders
   ValueSource(levelBytes int) encoding.ValueBuffer // source for the value decoder
   ```
   
   rep/def levels stay fully materialized (small); a streaming page's `Data()` returns nil. The streaming-vs-materialized branch lives inside these two methods, so the column reader stays branch-free.
   
   **Page reader.** Eligible pages skip `decompress()` in [`Next()`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/file/page_reader.go#L740). V2 levels are uncompressed with header byte-lengths; for V1 the rep|def region is peeled off the decompressed stream using exactly the length rule [`LevelDecoder.SetData`](https://github.com/apache/arrow-go/blob/d17f6870a0569b670d985e6eac60741af621bc88/parquet/internal/encoding/levels.go#L187) already uses (RLE 4-byte prefix / BIT_PACKED computed), leaving the stream at the first value byte. The value stream wraps an `io.LimitReader` sized to the compressed region; on page release it is drained and the decompressor closed, so the reader lands on the next header even if the caller skipped values.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.

zeroshade closed issue #865: [Go][Parquet] support streaming reads for large data pages
URL: https://github.com/apache/arrow-go/issues/865


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Go][Parquet] support streaming reads for large data pages [arrow-go]

Posted by "joechenrh (via GitHub)" <gi...@apache.org>.

joechenrh commented on issue #865:
URL: https://github.com/apache/arrow-go/issues/865#issuecomment-4828326689

   Yes, I'm still trying to figure out a clean way to implement this. 😸


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org