You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/10/17 17:41:53 UTC
[I] Pass options to Parquet metadata readers [arrow-rs]
etseidl opened a new issue, #8643:
URL: https://github.com/apache/arrow-rs/issues/8643
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- Part of #5853
One of the goals of the Thrift remodel project (#5854) was to enable such things as selective decoding of parts of the Parquet metadata. The parsers are now in place to enable this, but was is lacking now is a way to communicate what bits of the metadata are required.
**Describe the solution you'd like**
Some mechanism to communicate to the metadata parsers what is needed. Options can include such things as:
- Skip some statistics fields in `ColumnMetaData` (`Statistics`, `PageEncodingStatistics`, `SizeStatistics`, etc).
- Parse page encoding statistics into some other form (boolean, bitmask) to support dictionary based pushdown.
- Column projections (i.e. skip decoding metadata for columns that will not be read).
- Row group selection (only parse metadata for requested set of row groups).
- Only return schema.
- Skip schema and use a provided schema (perhaps from an earlier decode).
- Perhaps move encryption parameters here as well.
- Others I haven't yet thought of.
**Describe alternatives you've considered**
These options could be added to current properties objects, but there doesn't seem to b a single place for all of these. For instance, `SerializedFileReader` takes a `ReadOptions`, that contains a `ReaderProperties` which is what is subsequently used by the `SerialzedRowGroupReader` and children. On the arrow side we instead use an `ArrowReaderOptions`. The `ParquetMetaDataReader` and `ParquetMetaDataPushDecoder` manage their own set of options. It would be nice to have a single place to set metadata parsing options and then pass that to the respective decoders.
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464057986
Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.
For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?
I think the answer is yes....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3475799336
> > I don't quite understand this proposal
>
> I don't know should we separate to:
>
> 1. Decode all row-groups with limited columns ( or a selected set of row-groups, if compute engine has row-groups info in metadata), maybe only decode the filter columns min-max stats?
>
> 2. If we have all row-groups with limited columns, prune row-groups by statisitcs
>
> 3. Then, decode the Column Index by filter columns and limited row-groups, and fill it to `ParquetMetadata`?
That sounds right to me. Let's say in step 1 there's a predicate on column x and projection of columns y and z, so we decode column metadata for x with statistics, and y/z without statistics. In step 2 we apply the predicate on x to see if we can remove any row groups off the bat, and then in 3 we fetch the column index for x and offset index for xyz.
But we still need to address https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464051374
Maybe internally we change all the `Vec<XMetaData>` to `Vec<Option<XMetaData>>` so indexing still works, but what to do when an invalid index is passed if we don't change the API to return `Option` (although the current APIs don't seem to do bounds checking so they'll panic on bad indexes already)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3474909503
> I think there would be a interface or stage between load footer statics ( Like the code above, perhaps between DecodeState::ReadingPageIndex and DecodeState::Finished). Maybe add new statistics here is a good way for page index.
I don't quite understand this proposal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3475747548
> I don't quite understand this proposal
I don't know should we separate to:
1. Decode all row-groups with limited columns ( or a selected set of row-groups, if compute engine has row-groups info in metadata), maybe only decode the filter columns min-max stats?
2. If we have all row-groups with limited columns, prune row-groups by statisitcs
3. Then, decode the Column Index by filter columns and limited row-groups, and fill it to `ParquetMetadata`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3474058985
I plan to start on this with a single concrete use (like read only schema/reuse schema across files) just to get the plumbing in place. Then we can start adding more use cases, such as selective decoding of page indexes, then selective decoding of column chunk metadata. I imagine this will be part of the public API, so I don't want to add a bunch of speculative features up front, only to finding we need to change them later and having to make breaking changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3487600095
> Not a show stopper, but to avoid breaking changes we'll need to be careful. I'd imagine many of the options on the ArrowReaderBuilder (projection, row_groups, etc) would need to move to the metadata options, but be left in the current API with deprecation notices and a warning that setting such options with both the ArrowReaderBuilder and MetadataOptions may lead to UB.
I am not 100% sure we should consolidate the options
Specifically, I think the reader builders should be constructible with pre-existing metadata as well (e.g. when it was cached), so there needs to be some way to represent the "read only X row group" even when we already have the ParquetMetadata I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464051374
Another interesting API question: How do we represent the metadata for a subset of columns/row groups?
Right now `ParquetMetaData` has apis in terms of 'row group index' (and column index)
https://github.com/apache/arrow-rs/blob/e9ea12b138f8fa11c6046e94cb2b55b2830a0d7c/parquet/src/file/metadata/mod.rs#L243-L247
SO if we only got the RowGroupMetadata for 1 Row Group from a file with 10 Row Groups what would `metadata.num_row_groups()` return?
The same question applies to columns chunks too I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3474159198
As @etseidl tells me, the code below ( which is under parquet and not under arrow module ) would read the required row-group.
https://github.com/apache/arrow-rs/blob/bac0cb57af36f0c025696db146eccea8f3f469cb/parquet/src/file/serialized_reader.rs#L200-L216
> What I think would help is APIs for progressively reading / populating the metadata (e.g. initially only read 5 columns, but then be able to incrementally parse / produce the remaining columns after) -- this maybe is APIs on ParquetMetaData to add new columns, / row groups
I think there would be a interface or stage between load footer statics ( Like the code above, perhaps between `DecodeState::ReadingPageIndex` and `DecodeState::Finished`). Maybe add new statistics here is a good way for page index.
Besides, this might work like `ArrowPredicate` in arrow module: statistics/column index needs the `ArrowPredicate::projection` columns, and other required columns should read all offset index. I'm new here and I don't know would ArrowPredicate also be used here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464037915
> The ParquetMetaDataReader and ParquetMetaDataPushDecoder manage their own set of options.
I looked at the code and I think the only real options now are "should I read the page index":
* https://docs.rs/parquet/latest/src/parquet/file/metadata/reader.rs.html#66-75
* https://docs.rs/parquet/latest/src/parquet/file/metadata/push_decoder.rs.html#221-232
What if we made a new(!!!) options object like
```rust
struct ParquetMetadataOptions {
...
}
```
And then passed it to the `ParquetMetadataReader` (which now just wraps the `MetadataPushDecoder`)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3475113654
I've started on a bare bones implementation #8763. I'm still working out how high to push this up, but at least now we can start testing out some ideas with metadata skipping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3481980949
One thing I've noticed now that I'm far along on #8763 is that it will take a bit of retooling the arrow reader API to reduce unnecessary metadata decoding. Take the example of wanting to create an arrow reader with a column projection.
```rust
let mask = ProjectionMask::leaves(builder.parquet_schema(), [2]);
let builder = ParquetRecordBatchReaderBuilder::try_new(file).with_projection(mask);
builder.build()?
```
The `try_new` calls `try_new_with_options` using default options, which will read the file metadata in its entirety before calling `Self::new_with_metadata`. https://github.com/apache/arrow-rs/blob/b8a192696b8216878f247b6a2d8dd63a09558063/parquet/src/arrow/arrow_reader/mod.rs#L744-L752
Only after we already have the parquet metadata in hand will we actually set the projection mask on the builder. To get around this, we'll need the projection mask set in the `ArrowReaderOptions`, which really means in its contained `MetadataOptions`.
Not a show stopper, but to avoid breaking changes we'll need to be careful. I'd imagine many of the options on the `ArrowReaderBuilder` (projection, row_groups, etc) would need to move to the metadata options, but be left in the current API with deprecation notices and a warning that setting such options with both the `ArrowReaderBuilder` and `MetadataOptions` may lead to UB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464119414
> Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.
>
> For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?
>
> I think the answer is yes....
I think so as well.
I need to look back at old discussions, but IIRC there was a suggestion to stash the footer bytes, and then materialize bits of metadata on demand. With an index this now becomes possible. That could solve the "how do we index this" question.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3467310457
> > Finally, another interesting question is if the ArrowReader should try and minimize the metadata decoding on its own.
> > For example, if the reader is asked to read only 3 columns, and no other instruction is given for metadata, should it only decode the metadata for those three columns?
> > I think the answer is yes....
>
> I think so as well.
>
> I need to look back at old discussions, but IIRC there was a suggestion to stash the footer bytes, and then materialize bits of metadata on demand. With an index this now becomes possible. That could solve the "how do we index this" question.
>
> Edit: it was [@XiangpengHao](https://github.com/XiangpengHao) [#5855 (comment)](https://github.com/apache/arrow-rs/issues/5855#issuecomment-2154960257). With the index as part of the footer, the penalty when wanting to read the entire file goes away.
I think the code in arrow-rs / parquet-rs should just do the best with what it has and leave additional caching / optimization to other layers.
For example, in DataFusion, now the code already caches the entire ParquetMetaData (including column index) and passes it into the arrow-rs code for all columns in many cases, so adding additional caching in the parquet reader itself seems unecessary.
What I think would help is APIs for progressively reading / populating the metadata (e.g. initially only read 5 columns, but then be able to incrementally parse / produce the remaining columns after) -- this maybe is APIs on ParquetMetaData to add new columns, / row groups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3464086872
> > The ParquetMetaDataReader and ParquetMetaDataPushDecoder manage their own set of options.
>
> I looked at the code and I think the only real options now are "should I read the page index":
>
> * https://docs.rs/parquet/latest/src/parquet/file/metadata/reader.rs.html#66-75
>
> * https://docs.rs/parquet/latest/src/parquet/file/metadata/push_decoder.rs.html#221-232
>
>
> What if we made a new(!!!) options object like
>
> struct ParquetMetadataOptions {
> ...
> }
>
> And then passed it to the `ParquetMetadataReader` (which now just wraps the `MetadataPushDecoder`)?
That's what I was thinking...but then who creates it and where? I'm starting to trace back from the metadata parsers to their calling sites to see what might make the most sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pass options to Parquet metadata readers [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on issue #8643:
URL: https://github.com/apache/arrow-rs/issues/8643#issuecomment-3488605090
> >
>
> I am not 100% sure we should consolidate the options
>
> Specifically, I think the reader builders should be constructible with pre-existing metadata as well (e.g. when it was cached), so there needs to be some way to represent the "read only X row group" even when we already have the ParquetMetadata I think
Certainly not all of them, but some need to be pushed down to the metadata engines somehow. Perhaps we just wrap certain parts in `Arc` and share between the builders and the metadata options.
I don't think this needs to be solved today, either. There's still low hanging fruit to be gathered (skipping stats, etc) that will be enabled once we have *some* kind of options object like #8763.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org