You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/08/15 22:28:33 UTC
[PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
etseidl opened a new pull request, #8160:
URL: https://github.com/apache/arrow-rs/pull/8160
# Which issue does this PR close?
**Note: this targets a feature branch, not main**
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax.
- Part of #5854.
# Rationale for this change
Speed
# What changes are included in this PR?
Still a work in progress, but begins the process of converting page index parsing to the new thrift decoder.
# Are these changes tested?
This PR actually uses the new decoder when parsing the page indexes using the existing machinery. As such all tests involving the page indexes should apply to this code.
# Are there any user-facing changes?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3197722865
> >
>
> What drives the need to convert to array of structs? Is that the representation of the ColumnIndex in Rust or is it something about how the thrift is encoded?
Yes...parquet-rs takes the existing `ColumnIndex` which is a struct of arrays, each `num_pages` in length, and turns that into `num_pages` `PageIndex` objects contained in a `NativeIndex`, which is then encapsulated in an `Index` enum variant. While we're remodeling we could blow that up, but I think that would have a pretty big ripple effect downstream.
> As you say, perhaps we could keep around a Bytes with the byte statistics in it, and store an offset there (rather than copying into their own structure).
I'll try playing around with that and see if it helps.
Also, I think this came up before, but only materializing the column index for columns being filtered on rather than for the entire schema would certainly help. Selectively writing them would be useful as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#discussion_r2283054774
##########
parquet/src/file/page_index/index.rs:
##########
@@ -306,6 +308,75 @@ impl<T: ParquetValueType> NativeIndex<T> {
definition_level_histograms,
)
}
+
+ /// Creates a new [`NativeIndex`]
+ pub(crate) fn try_new_local(index: ColumnIndex) -> Result<Self, ParquetError> {
+ let len = index.min_values.len();
+
+ // turn Option<Vec<i64>> into Vec<Option<i64>>
+ let null_counts = index
+ .null_counts
+ .map(|x| x.into_iter().map(Some).collect::<Vec<_>>())
+ .unwrap_or_else(|| vec![None; len]);
+
+ // histograms are a 1D array encoding a 2D num_pages X num_levels matrix.
+ let to_page_histograms = |opt_hist: Option<Vec<i64>>| {
+ if let Some(hist) = opt_hist {
+ // TODO: should we assert (hist.len() % len) == 0?
+ let num_levels = hist.len() / len;
+ let mut res = Vec::with_capacity(len);
+ for i in 0..len {
+ let page_idx = i * num_levels;
+ let page_hist = hist[page_idx..page_idx + num_levels].to_vec();
+ res.push(Some(LevelHistogram::from(page_hist)));
+ }
+ res
+ } else {
+ vec![None; len]
+ }
+ };
+
+ // turn Option<Vec<i64>> into Vec<Option<i64>>
Review Comment:
😢
##########
parquet/src/file/page_index/offset_index.rs:
##########
@@ -55,24 +59,26 @@ impl From<&PageLocation> for crate::format::PageLocation {
}
}
+thrift_struct!(
/// [`OffsetIndex`] information for a column chunk. Contains offsets and sizes for each page
/// in the chunk. Optionally stores fully decoded page sizes for BYTE_ARRAY columns.
///
/// [`OffsetIndex`]: https://github.com/apache/parquet-format/blob/master/PageIndex.md
-#[derive(Debug, Clone, PartialEq)]
pub struct OffsetIndexMetaData {
- /// Vector of [`PageLocation`] objects, one per page in the chunk.
- pub page_locations: Vec<PageLocation>,
- /// Optional vector of unencoded page sizes, one per page in the chunk.
- /// Only defined for BYTE_ARRAY columns.
- pub unencoded_byte_array_data_bytes: Option<Vec<i64>>,
+ /// Vector of [`PageLocation`] objects, one per page in the chunk.
+ 1: required list<PageLocation> page_locations
Review Comment:
I will admit these macros are quite neat and are growing on me
##########
parquet/src/file/page_index/index_reader.rs:
##########
@@ -129,25 +130,37 @@ pub fn read_offset_indexes<R: ChunkReader>(
}
pub(crate) fn decode_offset_index(data: &[u8]) -> Result<OffsetIndexMetaData, ParquetError> {
- let mut prot = TCompactSliceInputProtocol::new(data);
- let offset = crate::format::OffsetIndex::read_from_in_protocol(&mut prot)?;
- OffsetIndexMetaData::try_new(offset)
+ let mut prot = ThriftCompactInputProtocol::new(data);
+ OffsetIndexMetaData::try_from(&mut prot)
}
-pub(crate) fn decode_column_index(data: &[u8], column_type: Type) -> Result<Index, ParquetError> {
- let mut prot = TCompactSliceInputProtocol::new(data);
+thrift_struct!(
+pub(crate) struct ColumnIndex<'a> {
+ 1: required list<bool> null_pages
+ 2: required list<'a><binary> min_values
Review Comment:
This is so sad -- the structure actually starts out column oriented and then we pivot it to rows (just to have to pivot it back to columns to use in DataFusion)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3197249445
> I will say that the page indexes are pretty darn expensive to parse, and the file used for the benchmark (`parquet-testing/data/all_types_tiny_pages.parquet`) is pretty pathological. Looking into where the time goes, the offset index is hobbled by the fact that it's defined as an array of structs, which adds considerable overhead to the parsing. The column index is a struct of arrays that parses very quickly, but then must be transformed into an array of structs after decoding, so that takes a good bit of time.
What drives the need to convert to array of structs? Is that the representation of the ColumnIndex in Rust or is it something about how the thrift is encoded?
Copying of the min/max statistics for byte arrays takes the majority of that time (note that the test file does not contain the level histograms...those would be very costly as well if present). We could look into rethinking how we represent the column index. Perhaps saving the bytes read and presenting slices rather than copies will work (at least as far as the histograms in the column index...we may be stuck with min/max value copying).
As you say, perhaps we could keep around a `Bytes` with the byte statistics in it, and store an offset there (rather than copying into their own structure).
Maybe we could also contemplate some way to defer decoding/copying the structures out until they were requested
> @alamb, not sure how radical you want to go here 😅
I have no pre-concieved ideas. I have personally always found the ColumnIndex representation in Rust (`Vec<Vec<Index>>` as I recall) quite complicated to work with, so if we have to change that to improve the performance I would be fully in support of it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3192922581
I will say that the page indexes are pretty darn expensive to parse, and the file used for the benchmark (`parquet-testing/data/all_types_tiny_pages.parquet`) is pretty pathological. Looking into where the time goes, the offset index is hobbled by the fact that it's defined as an array of structs, which adds considerable overhead to the parsing. The column index is a struct of arrays that parses very quickly, but then must be transformed into an array of structs after decoding, so that takes a good bit of time. Copying of the min/max statistics for byte arrays takes the majority of that time (note that the test file does not contain the level histograms...those would be very costly as well if present). We could look into rethinking how we represent the column index. Perhaps saving the bytes read and presenting slices rather than copies will work (at least as far as the histograms in the column index...we may be stuck with min/max value copying).
@alamb, not sure how radical you want to go here 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3197769153
> Also, I think this came up before, but only materializing the column index for columns being filtered on rather than for the entire schema would certainly help. Selectively writing them would be useful as well.
Yes, absolutely. Another really useful thing would be not decoding the page index / column index unless it is needed -- for example if we can prune the entire row group just with the row group statistics, we shouldn't even have to bother to decode the page index for that 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3192912281
Quick comparison of the old and new decoders:
```
old
open(page index) time: [1.7446 ms 1.7524 ms 1.7639 ms]
new
open(page index) time: [698.00 µs 699.75 µs 701.66 µs]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Decoding of page indexes [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8160:
URL: https://github.com/apache/arrow-rs/pull/8160#issuecomment-3207716481
Time to merge this one. Next steps include further speed up of the offset index decoding, and creating a new column index avatar that avoids the costly conversion to an array of structs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org