You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "etseidl (via GitHub)" <gi...@apache.org> on 2025/10/10 18:26:05 UTC
[PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
etseidl opened a new pull request, #8587:
URL: https://github.com/apache/arrow-rs/pull/8587
# Which issue does this PR close?
- Part of #5853
- Part of #8518.
# Rationale for this change
Reduce code clutter (too many `[cfg(feature = "encryption")]`s).
Also reduces the number of vector allocations using a strategy proposed by @jhorstmann (https://github.com/apache/arrow-rs/issues/8518#issuecomment-3355199067).
# What changes are included in this PR?
Refactors (most) of the Thrift decoding involving encryption into a new module.
Replaces the `encodings` vector in `ColumnChunkMetaData` with a bitmask.
# Are these changes tested?
Covered by existing and new tests.
# Are there any user-facing changes?
Yes. This changes the return type of `ColumnChunkMetaData::encodings` from `&Vec<Encoding>` to `Vec<Encoding>` (the vector is now created on demand rather than stored).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421869144
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
Review Comment:
- filed https://github.com/apache/arrow-rs/issues/8588
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2422744984
##########
parquet/src/basic.rs:
##########
@@ -724,6 +725,123 @@ impl FromStr for Encoding {
}
}
+/// A bitmask representing the [`Encoding`]s employed while encoding a Parquet column chunk.
+///
+/// The Parquet [`ColumnMetaData`] struct contains an array that indicates what encodings were
+/// used when writing that column chunk. For memory and performance reasons, this crate reduces
+/// that array to bitmask, where each bit position represents a different [`Encoding`]. This
+/// struct contains that bitmask, and provides methods to interact with the data.
+///
+/// # Example
+/// ```no_run
+/// # use parquet::file::metadata::ParquetMetaDataReader;
+/// # use parquet::basic::Encoding;
+/// # fn open_parquet_file(path: &str) -> std::fs::File { unimplemented!(); }
+/// // read parquet metadata from a file
+/// let file = open_parquet_file("some_path.parquet");
+/// let mut reader = ParquetMetaDataReader::new();
+/// reader.try_parse(&file).unwrap();
+/// let metadata = reader.finish().unwrap();
+///
+/// // find the encodings used by the first column chunk in the first row group
+/// let col_meta = metadata.row_group(0).column(0);
+/// let encodings = col_meta.encodings_mask();
+///
+/// // check to see if a particular encoding was used
+/// let used_rle = encodings.is_set(Encoding::RLE);
+///
+/// // check to see if all of a set of encodings were used
+/// let used_all = encodings.all_set([Encoding::RLE, Encoding::PLAIN].iter());
+///
+/// // convert mask to a Vec<Encoding>
+/// let encodings_vec = encodings.encodings().collect::<Vec<_>>();
+/// ```
+///
+/// [`ColumnMetaData`]: https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/src/main/thrift/parquet.thrift#L875
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct EncodingMask(i32);
Review Comment:
❤️
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1179,6 +1186,12 @@ impl ColumnChunkMetaDataBuilder {
/// Sets list of encodings for this column chunk.
pub fn set_encodings(mut self, encodings: Vec<Encoding>) -> Self {
Review Comment:
As a follow on PR it may be worth deprecating the old APIs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391947323
Here's a before and after look at the allocation burden. Getting rid of the page encoding stats allocation should have about the same benefit as this change.
Before
<img width="1066" height="308" alt="Screen Shot 2025-10-10 at 12 09 13 PM" src="https://github.com/user-attachments/assets/6a00350e-4201-4054-b3b2-f8a66b307496" />
After
<img width="1062" height="303" alt="Screen Shot 2025-10-10 at 12 04 13 PM" src="https://github.com/user-attachments/assets/fc7af7b4-0827-4b45-8f37-9e28d24686ef" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421737406
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
+ mask_to_encodings_vec(self.encodings)
+ }
+
+ /// All encodings used for this column, returned as a bitmask.
+ pub fn encodings_mask(&self) -> i32 {
Review Comment:
Thanks. Speed first, then usability 😆
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2423090284
##########
parquet/src/basic.rs:
##########
@@ -724,6 +725,123 @@ impl FromStr for Encoding {
}
}
+/// A bitmask representing the [`Encoding`]s employed while encoding a Parquet column chunk.
+///
+/// The Parquet [`ColumnMetaData`] struct contains an array that indicates what encodings were
+/// used when writing that column chunk. For memory and performance reasons, this crate reduces
+/// that array to bitmask, where each bit position represents a different [`Encoding`]. This
+/// struct contains that bitmask, and provides methods to interact with the data.
+///
+/// # Example
+/// ```no_run
+/// # use parquet::file::metadata::ParquetMetaDataReader;
+/// # use parquet::basic::Encoding;
+/// # fn open_parquet_file(path: &str) -> std::fs::File { unimplemented!(); }
+/// // read parquet metadata from a file
+/// let file = open_parquet_file("some_path.parquet");
+/// let mut reader = ParquetMetaDataReader::new();
+/// reader.try_parse(&file).unwrap();
+/// let metadata = reader.finish().unwrap();
+///
+/// // find the encodings used by the first column chunk in the first row group
+/// let col_meta = metadata.row_group(0).column(0);
+/// let encodings = col_meta.encodings_mask();
+///
+/// // check to see if a particular encoding was used
+/// let used_rle = encodings.is_set(Encoding::RLE);
+///
+/// // check to see if all of a set of encodings were used
+/// let used_all = encodings.all_set([Encoding::RLE, Encoding::PLAIN].iter());
+///
+/// // convert mask to a Vec<Encoding>
+/// let encodings_vec = encodings.encodings().collect::<Vec<_>>();
+/// ```
+///
+/// [`ColumnMetaData`]: https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/src/main/thrift/parquet.thrift#L875
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct EncodingMask(i32);
+
+impl EncodingMask {
+ const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
+
+ /// Create a new `EncodingMask` from an integer.
+ pub fn new(val: i32) -> Self {
+ Self(val)
+ }
+
+ /// Return an integer representation of this `EncodingMask`.
+ pub fn as_i32(&self) -> i32 {
+ self.0
+ }
+
+ /// Create a new `EncodingMask` from a collection of [`Encoding`]s.
+ pub fn new_from_encodings<'a>(encodings: impl Iterator<Item = &'a Encoding>) -> Self {
+ let mut mask = 0;
+ for &e in encodings {
+ mask |= 1 << (e as i32);
+ }
+ Self(mask)
+ }
+
+ /// Test if a given [`Encoding`] is present in this mask.
+ pub fn is_set(&self, val: Encoding) -> bool {
+ self.0 & (1 << (val as i32)) != 0
+ }
+
+ /// Test if all [`Encoding`]s in a given set are present in this mask.
+ pub fn all_set<'a>(&self, mut encodings: impl Iterator<Item = &'a Encoding>) -> bool {
+ encodings.all(|&e| self.is_set(e))
+ }
+
+ /// Return an iterator over all [`Encoding`]s present in this mask.
+ pub fn encodings(&self) -> impl Iterator<Item = Encoding> {
+ Self::mask_to_encodings_iter(self.0)
+ }
+
+ fn mask_to_encodings_iter(mask: i32) -> impl Iterator<Item = Encoding> {
+ (0..=Self::MAX_ENCODING)
+ .filter(move |i| mask & (1 << i) != 0)
+ .map(i32_to_encoding)
+ }
+}
+
+impl HeapSize for EncodingMask {
+ fn heap_size(&self) -> usize {
+ 0 // no heap allocations
+ }
+}
+
+impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for EncodingMask {
+ fn read_thrift(prot: &mut R) -> Result<Self> {
+ let mut mask = 0;
+
+ let list_ident = prot.read_list_begin()?;
+ for _ in 0..list_ident.size {
+ let val = i32::read_thrift(prot)?;
+ if (0..=Self::MAX_ENCODING).contains(&val) {
Review Comment:
Oh, and I was wondering if `i32` was too large, but we're already up to 10 bits with the current encodings, and with @alamb starting the discussion about adding more, I think we're likely to exhaust an `i16` in the not-to-distant future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391726888
This PR shows the value of reducing vector allocations. Two major offenders during decoding are the `encodings` and `page_encoding_stats` vectors in `ColumnChunkMetaData`. This PR gets rid of the former. The biggest impact is in the wide schema test.
```
default features
group encodings_default main
----- ----------------- -------
decode parquet metadata 1.00 16.2±0.28µs ? ?/sec 1.08 17.4±0.31µs ? ?/sec
decode parquet metadata (wide) 1.00 62.0±1.92ms ? ?/sec 1.20 74.2±1.58ms ? ?/sec
open(default) 1.00 16.9±0.25µs ? ?/sec 1.07 18.1±0.41µs ? ?/sec
open(page index) 1.01 255.8±6.08µs ? ?/sec 1.00 252.9±4.98µs ? ?/sec
all features
group encodings_encr main
----- -------------- ------------
decode parquet metadata 1.00 16.9±0.29µs ? ?/sec 1.07 18.0±0.32µs ? ?/sec
decode parquet metadata (wide) 1.00 67.6±1.37ms ? ?/sec 1.16 78.5±1.69ms ? ?/sec
open(default) 1.00 17.5±0.35µs ? ?/sec 1.07 18.8±0.39µs ? ?/sec
open(page index) 1.00 254.1±4.33µs ? ?/sec 1.00 254.6±5.31µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421729591
##########
parquet/src/file/metadata/encryption.rs:
##########
Review Comment:
Sounds good. And rename `thrift_gen` to just `thrift`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421743641
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
+
+/// Given a Thrift input stream, read a vector of [`Encoding`] enums and convert to a bitmask.
+pub(super) fn thrift_encodings_to_mask<'a, R: ThriftCompactInputProtocol<'a>>(
+ prot: &mut R,
+) -> Result<i32> {
+ let mut mask = 0;
+
+ let list_ident = prot.read_list_begin()?;
+ for _ in 0..list_ident.size {
+ let val = i32::read_thrift(prot)?;
+ if (0..=MAX_ENCODING).contains(&val) {
+ mask |= 1 << val;
+ }
+ }
+ Ok(mask)
+}
+
+#[allow(deprecated)]
Review Comment:
`Encoding::BIT_PACKED` is deprecated (but we keep it because it's still in the spec, but marked as deprecated there as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421732797
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
Review Comment:
Sounds good. I started down that path before but got lazy 😅. I'll add it to the queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3398798189
Merging this to move on to some refactoring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421652450
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
Review Comment:
This is the breaking change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391736884
Note to reviewers: this PR is actually two things, a refactor and a performance enhancement. I've left them as two commits to make it easier to see what changed in each part.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421746759
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
Review Comment:
will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3392048329
The wide results are surprising (and disappointing 😢). I'll test on a better machine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421723973
##########
parquet/src/file/metadata/encryption.rs:
##########
@@ -0,0 +1,544 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// a collection of generated structs used to parse thrift metadata
Review Comment:
True. Also this is just cut-and-paste, so I should update it to match what's actually in the module.
Also, we (I) need to rename `thrift_gen` at some point, since that's not strictly true either.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391766866
> This PR shows the value of reducing vector allocations. Two major offenders during decoding are the encodings and page_encoding_stats vectors in ColumnChunkMetaData. This PR gets rid of the former. The biggest impact is in the wide schema test.
I have a good feeling that this thrift decoder is going to be the gift that keeps on giving performance wise 🎁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421742231
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
Review Comment:
shall I file a ticket? (I personally get an endorphin hit for each ticket that I close, so want to offer you the same incentive 😆 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391778446
> I have a good feeling that this thrift decoder is going to be the gift that keeps on giving performance wise 🎁
There *is* more to come 😁
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3391827414
I also queued up a benchmark run to confirm the performance improvement (I am sure it is there!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3392016099
🤖 `./gh_compare_arrow.sh` [Benchmark Script](https://github.com/alamb/datafusion-benchmarking/blob/main/scripts/gh_compare_arrow.sh) Running
Linux aal-dev 6.14.0-1016-gcp #17~24.04.1-Ubuntu SMP Wed Sep 3 01:55:36 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing encryption_and_encodings (a88699526d945f60d95179b3254d25609980ee31) to b51a0003ad13bc91245e350a9e8d9beaa3c8b35f [diff](https://github.com/apache/arrow-rs/compare/b51a0003ad13bc91245e350a9e8d9beaa3c8b35f..a88699526d945f60d95179b3254d25609980ee31)
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental --bench metadata
BENCH_FILTER=
BENCH_BRANCH_NAME=encryption_and_encodings
Results will be posted here when complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3392029236
🤖: Benchmark completed
<details><summary>Details</summary>
<p>
```
group encryption_and_encodings main
----- ------------------------ ----
decode parquet metadata 1.00 11.0±0.03µs ? ?/sec 1.08 12.0±0.03µs ? ?/sec
decode parquet metadata (wide) 1.00 60.6±7.84ms ? ?/sec 1.01 61.2±0.85ms ? ?/sec
open(default) 1.00 10.8±0.28µs ? ?/sec 1.07 11.6±0.04µs ? ?/sec
open(page index) 1.00 201.1±0.91µs ? ?/sec 1.00 201.5±0.67µs ? ?/sec
```
</p>
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421890102
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
Review Comment:
Done in https://github.com/apache/arrow-rs/pull/8587/commits/c94c263b6e8a1e72738851d8571fb409b9fd1a91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2422164474
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
Review Comment:
First stab in https://github.com/apache/arrow-rs/pull/8587/commits/591d1ec6aa3fa1bcef6750e4f95b047a1d4a6305
@alamb please take a look and see if this is on the right track
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2422812018
##########
parquet/src/basic.rs:
##########
@@ -724,6 +725,123 @@ impl FromStr for Encoding {
}
}
+/// A bitmask representing the [`Encoding`]s employed while encoding a Parquet column chunk.
+///
+/// The Parquet [`ColumnMetaData`] struct contains an array that indicates what encodings were
+/// used when writing that column chunk. For memory and performance reasons, this crate reduces
+/// that array to bitmask, where each bit position represents a different [`Encoding`]. This
+/// struct contains that bitmask, and provides methods to interact with the data.
+///
+/// # Example
+/// ```no_run
+/// # use parquet::file::metadata::ParquetMetaDataReader;
+/// # use parquet::basic::Encoding;
+/// # fn open_parquet_file(path: &str) -> std::fs::File { unimplemented!(); }
+/// // read parquet metadata from a file
+/// let file = open_parquet_file("some_path.parquet");
+/// let mut reader = ParquetMetaDataReader::new();
+/// reader.try_parse(&file).unwrap();
+/// let metadata = reader.finish().unwrap();
+///
+/// // find the encodings used by the first column chunk in the first row group
+/// let col_meta = metadata.row_group(0).column(0);
+/// let encodings = col_meta.encodings_mask();
+///
+/// // check to see if a particular encoding was used
+/// let used_rle = encodings.is_set(Encoding::RLE);
+///
+/// // check to see if all of a set of encodings were used
+/// let used_all = encodings.all_set([Encoding::RLE, Encoding::PLAIN].iter());
+///
+/// // convert mask to a Vec<Encoding>
+/// let encodings_vec = encodings.encodings().collect::<Vec<_>>();
+/// ```
+///
+/// [`ColumnMetaData`]: https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/src/main/thrift/parquet.thrift#L875
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct EncodingMask(i32);
+
+impl EncodingMask {
+ const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
+
+ /// Create a new `EncodingMask` from an integer.
+ pub fn new(val: i32) -> Self {
+ Self(val)
+ }
+
+ /// Return an integer representation of this `EncodingMask`.
+ pub fn as_i32(&self) -> i32 {
+ self.0
+ }
+
+ /// Create a new `EncodingMask` from a collection of [`Encoding`]s.
+ pub fn new_from_encodings<'a>(encodings: impl Iterator<Item = &'a Encoding>) -> Self {
+ let mut mask = 0;
+ for &e in encodings {
+ mask |= 1 << (e as i32);
+ }
+ Self(mask)
+ }
+
+ /// Test if a given [`Encoding`] is present in this mask.
+ pub fn is_set(&self, val: Encoding) -> bool {
+ self.0 & (1 << (val as i32)) != 0
+ }
+
+ /// Test if all [`Encoding`]s in a given set are present in this mask.
+ pub fn all_set<'a>(&self, mut encodings: impl Iterator<Item = &'a Encoding>) -> bool {
+ encodings.all(|&e| self.is_set(e))
+ }
+
+ /// Return an iterator over all [`Encoding`]s present in this mask.
+ pub fn encodings(&self) -> impl Iterator<Item = Encoding> {
+ Self::mask_to_encodings_iter(self.0)
+ }
+
+ fn mask_to_encodings_iter(mask: i32) -> impl Iterator<Item = Encoding> {
+ (0..=Self::MAX_ENCODING)
+ .filter(move |i| mask & (1 << i) != 0)
+ .map(i32_to_encoding)
+ }
+}
+
+impl HeapSize for EncodingMask {
+ fn heap_size(&self) -> usize {
+ 0 // no heap allocations
+ }
+}
+
+impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for EncodingMask {
+ fn read_thrift(prot: &mut R) -> Result<Self> {
+ let mut mask = 0;
+
+ let list_ident = prot.read_list_begin()?;
+ for _ in 0..list_ident.size {
+ let val = i32::read_thrift(prot)?;
+ if (0..=Self::MAX_ENCODING).contains(&val) {
Review Comment:
I've been thinking about this representation some more since I proposed it. The main usecase is to detect whether any used encoding is not yet supported. I think this means we also need to handle bits outside the current `MAX_ENCODING`, and also iterate over all set bits in the `encodings` method. It's very unlikely there will ever be over 32 encodings, but maybe we should also clamp the shift value to 31 to avoid any overflow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl merged PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2423089770
##########
parquet/src/basic.rs:
##########
@@ -724,6 +725,123 @@ impl FromStr for Encoding {
}
}
+/// A bitmask representing the [`Encoding`]s employed while encoding a Parquet column chunk.
+///
+/// The Parquet [`ColumnMetaData`] struct contains an array that indicates what encodings were
+/// used when writing that column chunk. For memory and performance reasons, this crate reduces
+/// that array to bitmask, where each bit position represents a different [`Encoding`]. This
+/// struct contains that bitmask, and provides methods to interact with the data.
+///
+/// # Example
+/// ```no_run
+/// # use parquet::file::metadata::ParquetMetaDataReader;
+/// # use parquet::basic::Encoding;
+/// # fn open_parquet_file(path: &str) -> std::fs::File { unimplemented!(); }
+/// // read parquet metadata from a file
+/// let file = open_parquet_file("some_path.parquet");
+/// let mut reader = ParquetMetaDataReader::new();
+/// reader.try_parse(&file).unwrap();
+/// let metadata = reader.finish().unwrap();
+///
+/// // find the encodings used by the first column chunk in the first row group
+/// let col_meta = metadata.row_group(0).column(0);
+/// let encodings = col_meta.encodings_mask();
+///
+/// // check to see if a particular encoding was used
+/// let used_rle = encodings.is_set(Encoding::RLE);
+///
+/// // check to see if all of a set of encodings were used
+/// let used_all = encodings.all_set([Encoding::RLE, Encoding::PLAIN].iter());
+///
+/// // convert mask to a Vec<Encoding>
+/// let encodings_vec = encodings.encodings().collect::<Vec<_>>();
+/// ```
+///
+/// [`ColumnMetaData`]: https://github.com/apache/parquet-format/blob/9fd57b59e0ce1a82a69237dcf8977d3e72a2965d/src/main/thrift/parquet.thrift#L875
+#[derive(Debug, Clone, Default, PartialEq, Eq)]
+pub struct EncodingMask(i32);
+
+impl EncodingMask {
+ const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
+
+ /// Create a new `EncodingMask` from an integer.
+ pub fn new(val: i32) -> Self {
+ Self(val)
+ }
+
+ /// Return an integer representation of this `EncodingMask`.
+ pub fn as_i32(&self) -> i32 {
+ self.0
+ }
+
+ /// Create a new `EncodingMask` from a collection of [`Encoding`]s.
+ pub fn new_from_encodings<'a>(encodings: impl Iterator<Item = &'a Encoding>) -> Self {
+ let mut mask = 0;
+ for &e in encodings {
+ mask |= 1 << (e as i32);
+ }
+ Self(mask)
+ }
+
+ /// Test if a given [`Encoding`] is present in this mask.
+ pub fn is_set(&self, val: Encoding) -> bool {
+ self.0 & (1 << (val as i32)) != 0
+ }
+
+ /// Test if all [`Encoding`]s in a given set are present in this mask.
+ pub fn all_set<'a>(&self, mut encodings: impl Iterator<Item = &'a Encoding>) -> bool {
+ encodings.all(|&e| self.is_set(e))
+ }
+
+ /// Return an iterator over all [`Encoding`]s present in this mask.
+ pub fn encodings(&self) -> impl Iterator<Item = Encoding> {
+ Self::mask_to_encodings_iter(self.0)
+ }
+
+ fn mask_to_encodings_iter(mask: i32) -> impl Iterator<Item = Encoding> {
+ (0..=Self::MAX_ENCODING)
+ .filter(move |i| mask & (1 << i) != 0)
+ .map(i32_to_encoding)
+ }
+}
+
+impl HeapSize for EncodingMask {
+ fn heap_size(&self) -> usize {
+ 0 // no heap allocations
+ }
+}
+
+impl<'a, R: ThriftCompactInputProtocol<'a>> ReadThrift<'a, R> for EncodingMask {
+ fn read_thrift(prot: &mut R) -> Result<Self> {
+ let mut mask = 0;
+
+ let list_ident = prot.read_list_begin()?;
+ for _ in 0..list_ident.size {
+ let val = i32::read_thrift(prot)?;
+ if (0..=Self::MAX_ENCODING).contains(&val) {
Review Comment:
Thanks @jhorstmann, that's an excellent point. I think for each enum/union we should figure out what we want to do when we encounter an unknown field. For now they all (with the exception of `LogicalType`, `EdgeInerpolationAlgorithm`, and `ColumnOrder`) return an error.
`Encoding` likely *should* return an error since the column will not be decodable. I think I'll change the line above to `Encoding::read_thrift` which will detect the error so we don't have to check for it here. The test on the range can then be removed as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421649871
##########
parquet/src/file/metadata/encryption.rs:
##########
@@ -0,0 +1,544 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// a collection of generated structs used to parse thrift metadata
+
+use std::io::Write;
+
+use crate::{
+ basic::{Compression, thrift_encodings_to_mask},
+ encryption::decrypt::{FileDecryptionProperties, FileDecryptor},
+ errors::{ParquetError, Result},
+ file::{
+ column_crypto_metadata::ColumnCryptoMetaData,
+ metadata::{
+ HeapSize, LevelHistogram, PageEncodingStats, ParquetMetaData, RowGroupMetaData,
+ thrift_gen::{
+ GeospatialStatistics, SizeStatistics, Statistics, convert_geo_stats, convert_stats,
+ parquet_metadata_from_bytes,
+ },
+ },
+ },
+ parquet_thrift::{
+ ElementType, FieldType, ReadThrift, ThriftCompactInputProtocol,
+ ThriftCompactOutputProtocol, ThriftSliceInputProtocol, WriteThrift, WriteThriftField,
+ read_thrift_vec,
+ },
+ thrift_struct, thrift_union,
+};
+
+thrift_struct!(
+pub(crate) struct AesGcmV1 {
+ /// AAD prefix
+ 1: optional binary aad_prefix
+
+ /// Unique file identifier part of AAD suffix
+ 2: optional binary aad_file_unique
+
+ /// In files encrypted with AAD prefix without storing it,
+ /// readers must supply the prefix
+ 3: optional bool supply_aad_prefix
+}
+);
+
+impl HeapSize for AesGcmV1 {
+ fn heap_size(&self) -> usize {
+ self.aad_prefix.heap_size()
+ + self.aad_file_unique.heap_size()
+ + self.supply_aad_prefix.heap_size()
+ }
+}
+
+thrift_struct!(
+pub(crate) struct AesGcmCtrV1 {
+ /// AAD prefix
+ 1: optional binary aad_prefix
+
+ /// Unique file identifier part of AAD suffix
+ 2: optional binary aad_file_unique
+
+ /// In files encrypted with AAD prefix without storing it,
+ /// readers must supply the prefix
+ 3: optional bool supply_aad_prefix
+}
+);
+
+impl HeapSize for AesGcmCtrV1 {
+ fn heap_size(&self) -> usize {
+ self.aad_prefix.heap_size()
+ + self.aad_file_unique.heap_size()
+ + self.supply_aad_prefix.heap_size()
+ }
+}
+
+thrift_union!(
+union EncryptionAlgorithm {
+ 1: (AesGcmV1) AES_GCM_V1
+ 2: (AesGcmCtrV1) AES_GCM_CTR_V1
+}
+);
+
+impl HeapSize for EncryptionAlgorithm {
+ fn heap_size(&self) -> usize {
+ match self {
+ Self::AES_GCM_V1(gcm) => gcm.heap_size(),
+ Self::AES_GCM_CTR_V1(gcm_ctr) => gcm_ctr.heap_size(),
+ }
+ }
+}
+
+thrift_struct!(
+/// Crypto metadata for files with encrypted footer
+pub(crate) struct FileCryptoMetaData<'a> {
+ /// Encryption algorithm. This field is only used for files
+ /// with encrypted footer. Files with plaintext footer store algorithm id
+ /// inside footer (FileMetaData structure).
+ 1: required EncryptionAlgorithm encryption_algorithm
+
+ /// Retrieval metadata of key used for encryption of footer,
+ /// and (possibly) columns.
+ 2: optional binary<'a> key_metadata
+}
+);
+
+fn row_group_from_encrypted_thrift(
+ mut rg: RowGroupMetaData,
+ decryptor: Option<&FileDecryptor>,
+) -> Result<RowGroupMetaData> {
+ let schema_descr = rg.schema_descr;
+
+ if schema_descr.num_columns() != rg.columns.len() {
+ return Err(general_err!(
+ "Column count mismatch. Schema has {} columns while Row Group has {}",
+ schema_descr.num_columns(),
+ rg.columns.len()
+ ));
+ }
+ let total_byte_size = rg.total_byte_size;
+ let num_rows = rg.num_rows;
+ let mut columns = vec![];
+
+ for (i, (mut c, d)) in rg
+ .columns
+ .drain(0..)
+ .zip(schema_descr.columns())
+ .enumerate()
+ {
+ // Read encrypted metadata if it's present and we have a decryptor.
+ if let (true, Some(decryptor)) = (c.encrypted_column_metadata.is_some(), decryptor) {
+ let column_decryptor = match c.crypto_metadata() {
+ None => {
+ return Err(general_err!(
+ "No crypto_metadata is set for column '{}', which has encrypted metadata",
+ d.path().string()
+ ));
+ }
+ Some(ColumnCryptoMetaData::ENCRYPTION_WITH_COLUMN_KEY(crypto_metadata)) => {
+ let column_name = crypto_metadata.path_in_schema.join(".");
+ decryptor.get_column_metadata_decryptor(
+ column_name.as_str(),
+ crypto_metadata.key_metadata.as_deref(),
+ )?
+ }
+ Some(ColumnCryptoMetaData::ENCRYPTION_WITH_FOOTER_KEY) => {
+ decryptor.get_footer_decryptor()?
+ }
+ };
+
+ let column_aad = crate::encryption::modules::create_module_aad(
+ decryptor.file_aad(),
+ crate::encryption::modules::ModuleType::ColumnMetaData,
+ rg.ordinal.unwrap() as usize,
+ i,
+ None,
+ )?;
+
+ // Take the encrypted column metadata as it is no longer needed.
+ let encrypted_column_metadata = c.encrypted_column_metadata.take();
+ let buf = encrypted_column_metadata.unwrap();
+ let decrypted_cc_buf = column_decryptor
+ .decrypt(&buf, column_aad.as_ref())
+ .map_err(|_| {
+ general_err!(
+ "Unable to decrypt column '{}', perhaps the column key is wrong?",
+ d.path().string()
+ )
+ })?;
+
+ // parse decrypted buffer and then replace fields in 'c'
+ let col_meta = read_column_metadata(decrypted_cc_buf.as_slice())?;
+
+ let (
+ unencoded_byte_array_data_bytes,
+ repetition_level_histogram,
+ definition_level_histogram,
+ ) = if let Some(size_stats) = col_meta.size_statistics {
+ (
+ size_stats.unencoded_byte_array_data_bytes,
+ size_stats.repetition_level_histogram,
+ size_stats.definition_level_histogram,
+ )
+ } else {
+ (None, None, None)
+ };
+
+ let repetition_level_histogram = repetition_level_histogram.map(LevelHistogram::from);
+ let definition_level_histogram = definition_level_histogram.map(LevelHistogram::from);
+
+ c.encodings = col_meta.encodings;
+ c.compression = col_meta.codec;
+ c.num_values = col_meta.num_values;
+ c.total_uncompressed_size = col_meta.total_uncompressed_size;
+ c.total_compressed_size = col_meta.total_compressed_size;
+ c.data_page_offset = col_meta.data_page_offset;
+ c.index_page_offset = col_meta.index_page_offset;
+ c.dictionary_page_offset = col_meta.dictionary_page_offset;
+ c.statistics = convert_stats(d.physical_type(), col_meta.statistics)?;
+ c.encoding_stats = col_meta.encoding_stats;
+ c.bloom_filter_offset = col_meta.bloom_filter_offset;
+ c.bloom_filter_length = col_meta.bloom_filter_length;
+ c.unencoded_byte_array_data_bytes = unencoded_byte_array_data_bytes;
+ c.repetition_level_histogram = repetition_level_histogram;
+ c.definition_level_histogram = definition_level_histogram;
+ c.geo_statistics = convert_geo_stats(col_meta.geospatial_statistics);
+
+ columns.push(c);
+ } else {
+ columns.push(c);
+ }
+ }
+
+ let sorting_columns = rg.sorting_columns;
+ let file_offset = rg.file_offset;
+ let ordinal = rg.ordinal;
+
+ Ok(RowGroupMetaData {
+ columns,
+ num_rows,
+ sorting_columns,
+ total_byte_size,
+ schema_descr,
+ file_offset,
+ ordinal,
+ })
+}
+
+/// Decodes [`ParquetMetaData`] from the provided bytes, handling metadata that may be encrypted.
+///
+/// Typically this is used to decode the metadata from the end of a parquet
+/// file. The format of `buf` is the Thrift compact binary protocol, as specified
+/// by the [Parquet Spec]. Buffer can be encrypted with AES GCM or AES CTR
+/// ciphers as specfied in the [Parquet Encryption Spec].
+///
+/// [Parquet Spec]: https://github.com/apache/parquet-format#metadata
+/// [Parquet Encryption Spec]: https://parquet.apache.org/docs/file-format/data-pages/encryption/
+pub(crate) fn parquet_metadata_with_encryption(
+ file_decryption_properties: Option<&FileDecryptionProperties>,
+ encrypted_footer: bool,
+ buf: &[u8],
+) -> Result<ParquetMetaData> {
+ use crate::file::metadata::ParquetMetaDataBuilder;
+
+ let mut buf = buf;
+ let mut file_decryptor = None;
+ let decrypted_fmd_buf;
+
+ if encrypted_footer {
+ let mut prot = ThriftSliceInputProtocol::new(buf);
+ if let Some(file_decryption_properties) = file_decryption_properties {
+ let t_file_crypto_metadata: FileCryptoMetaData =
+ FileCryptoMetaData::read_thrift(&mut prot)
+ .map_err(|e| general_err!("Could not parse crypto metadata: {}", e))?;
+ let supply_aad_prefix = match &t_file_crypto_metadata.encryption_algorithm {
+ EncryptionAlgorithm::AES_GCM_V1(algo) => algo.supply_aad_prefix,
+ _ => Some(false),
+ }
+ .unwrap_or(false);
+ if supply_aad_prefix && file_decryption_properties.aad_prefix().is_none() {
+ return Err(general_err!(
+ "Parquet file was encrypted with an AAD prefix that is not stored in the file, \
+ but no AAD prefix was provided in the file decryption properties"
+ ));
+ }
+ let decryptor = get_file_decryptor(
+ t_file_crypto_metadata.encryption_algorithm,
+ t_file_crypto_metadata.key_metadata,
+ file_decryption_properties,
+ )?;
+ let footer_decryptor = decryptor.get_footer_decryptor();
+ let aad_footer = crate::encryption::modules::create_footer_aad(decryptor.file_aad())?;
+
+ decrypted_fmd_buf = footer_decryptor?
+ .decrypt(prot.as_slice().as_ref(), aad_footer.as_ref())
+ .map_err(|_| {
+ general_err!(
+ "Provided footer key and AAD were unable to decrypt parquet footer"
+ )
+ })?;
+
+ buf = &decrypted_fmd_buf;
+ file_decryptor = Some(decryptor);
+ } else {
+ return Err(general_err!(
+ "Parquet file has an encrypted footer but decryption properties were not provided"
+ ));
+ }
+ }
+
+ let parquet_meta = parquet_metadata_from_bytes(buf)
+ .map_err(|e| general_err!("Could not parse metadata: {}", e))?;
+
+ let ParquetMetaData {
+ mut file_metadata,
+ row_groups,
+ column_index: _,
+ offset_index: _,
+ file_decryptor: _,
+ } = parquet_meta;
+
+ // Take the encryption algorithm and footer signing key metadata as they are no longer
+ // needed after this.
+ if let (Some(algo), Some(file_decryption_properties)) = (
+ file_metadata.encryption_algorithm.take(),
+ file_decryption_properties,
+ ) {
+ let footer_signing_key_metadata = file_metadata.footer_signing_key_metadata.take();
+
+ // File has a plaintext footer but encryption algorithm is set
+ let file_decryptor_value = get_file_decryptor(
+ *algo,
+ footer_signing_key_metadata.as_deref(),
+ file_decryption_properties,
+ )?;
+ if file_decryption_properties.check_plaintext_footer_integrity() && !encrypted_footer {
+ file_decryptor_value.verify_plaintext_footer_signature(buf)?;
+ }
+ file_decryptor = Some(file_decryptor_value);
+ }
+
+ // decrypt column chunk info
+ let row_groups = row_groups
+ .into_iter()
+ .map(|rg| row_group_from_encrypted_thrift(rg, file_decryptor.as_ref()))
+ .collect::<Result<Vec<_>>>()?;
+
+ let metadata = ParquetMetaDataBuilder::new(file_metadata)
+ .set_row_groups(row_groups)
+ .set_file_decryptor(file_decryptor)
+ .build();
+
+ Ok(metadata)
+}
+
+fn get_file_decryptor(
+ encryption_algorithm: EncryptionAlgorithm,
+ footer_key_metadata: Option<&[u8]>,
+ file_decryption_properties: &FileDecryptionProperties,
+) -> Result<FileDecryptor> {
+ match encryption_algorithm {
+ EncryptionAlgorithm::AES_GCM_V1(algo) => {
+ let aad_file_unique = algo
+ .aad_file_unique
+ .ok_or_else(|| general_err!("AAD unique file identifier is not set"))?;
+ let aad_prefix = if let Some(aad_prefix) = file_decryption_properties.aad_prefix() {
+ aad_prefix.clone()
+ } else {
+ algo.aad_prefix.map(|v| v.to_vec()).unwrap_or_default()
+ };
+ let aad_file_unique = aad_file_unique.to_vec();
+
+ FileDecryptor::new(
+ file_decryption_properties,
+ footer_key_metadata,
+ aad_file_unique,
+ aad_prefix,
+ )
+ }
+ EncryptionAlgorithm::AES_GCM_CTR_V1(_) => Err(nyi_err!(
+ "The AES_GCM_CTR_V1 encryption algorithm is not yet supported"
+ )),
+ }
+}
+
+#[derive(Clone, Debug, Eq, PartialEq)]
+struct ColumnMetaData<'a> {
Review Comment:
Most of this diff is code moved here from `thrift_gen.rs`. From this point on is code added to support the change to `encodings`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#discussion_r2421667109
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -1148,7 +1155,7 @@ impl ColumnChunkMetaDataBuilder {
fn new(column_descr: ColumnDescPtr) -> Self {
Self(ColumnChunkMetaData {
column_descr,
- encodings: Vec::new(),
+ encodings: 0,
Review Comment:
No allocations

##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
+
+/// Given a Thrift input stream, read a vector of [`Encoding`] enums and convert to a bitmask.
+pub(super) fn thrift_encodings_to_mask<'a, R: ThriftCompactInputProtocol<'a>>(
+ prot: &mut R,
+) -> Result<i32> {
+ let mut mask = 0;
+
+ let list_ident = prot.read_list_begin()?;
+ for _ in 0..list_ident.size {
+ let val = i32::read_thrift(prot)?;
+ if (0..=MAX_ENCODING).contains(&val) {
+ mask |= 1 << val;
+ }
+ }
+ Ok(mask)
+}
+
+#[allow(deprecated)]
Review Comment:
what is deprecated about this?
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
Review Comment:
I also recommend changing the signature to avoid *forcing* an allocation -- for example how about returning an iterator like this?
```suggestion
pub fn encodings(&self) -> impl Iterator<Item = &Encoding> {
```
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
+ mask_to_encodings_vec(self.encodings)
+ }
+
+ /// All encodings used for this column, returned as a bitmask.
+ pub fn encodings_mask(&self) -> i32 {
Review Comment:
I think this will be very hard to use as a user given the lack of documentation / export of how to interpret the values.
I left a suggestion above about how to improve it
##########
parquet/src/file/metadata/encryption.rs:
##########
@@ -0,0 +1,544 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// a collection of generated structs used to parse thrift metadata
Review Comment:
technically speaking they aren't generated anymore
##########
parquet/src/basic.rs:
##########
@@ -724,6 +724,58 @@ impl FromStr for Encoding {
}
}
+const MAX_ENCODING: i32 = Encoding::BYTE_STREAM_SPLIT as i32;
Review Comment:
It is probably my OOP / old age, but I would find this API easier to deal with if it were wrapped in a type that had a documented API
Something like
```rust
struct EncodingMask(i32);
impl EncodingMask {
fn new(val: i32) -> Self {
...
}
fn as_i32(&self) -> i32 {
self.0
}
}
// then you can implement From impls to/from i32 as well as a no allocation iterator
```
I am happy to help hack this out if you like the idea
##########
parquet/src/file/metadata/mod.rs:
##########
@@ -968,8 +970,13 @@ impl ColumnChunkMetaData {
}
/// All encodings used for this column.
- pub fn encodings(&self) -> &Vec<Encoding> {
- &self.encodings
+ pub fn encodings(&self) -> Vec<Encoding> {
Review Comment:
How about we also mark this function as `#[deprecated]` to help people know they should use the more efficient `encoding_mask` if they are using it?
##########
parquet/src/file/metadata/encryption.rs:
##########
Review Comment:
Since this is thrift parsing related, perhaps it would be better organized as a submodule of thrift_get?
In other words, how about renaming it into `parquet/src/file/metadata/thrift_gen/encryption.rs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3392043189
Bencmark results look consistent with yours (though I didn't run with just default features)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [thrift-remodel] Refactor Thrift encryption and store encodings as bitmask [arrow-rs]
Posted by "etseidl (via GitHub)" <gi...@apache.org>.
etseidl commented on PR #8587:
URL: https://github.com/apache/arrow-rs/pull/8587#issuecomment-3392457638
Benchmark on my workstation makes me feel a little better
```
group encodings_encr main
----- -------------- -------
decode parquet metadata 1.00 5.9±0.06µs ? ?/sec 1.07 6.3±0.05µs ? ?/sec
decode parquet metadata (wide) 1.00 25.4±0.48ms ? ?/sec 1.10 27.8±1.08ms ? ?/sec
open(default) 1.00 6.2±0.10µs ? ?/sec 1.05 6.5±0.06µs ? ?/sec
open(page index) 1.00 107.0±1.30µs ? ?/sec 1.01 108.3±1.11µs ? ?/sec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org