You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "rok (via GitHub)" <gi...@apache.org> on 2025/03/13 12:51:20 UTC
[PR] Add example for how to read encrypted parquet files [arrow-rs]
rok opened a new pull request, #7283:
URL: https://github.com/apache/arrow-rs/pull/7283
# Which issue does this PR close?
Closes #7281.
# Rationale for this change
This is to provide users a starting point and a reference for reading encrypted files.
# What changes are included in this PR?
We add couple of examples to the parquet docs.
# Are there any user-facing changes?
Only in docs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1996251793
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,55 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+//! Note: This requires the experimental `encryption` feature to be enabled at compile time.
+//!
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // Define the AES encryption keys required required for decrypting the footer metadata
+//! // and column-specific data. If only a footer key is used then it is assumed that the
+//! // file uses uniform encryption and all columns are encrypted with the footer key.
+//! // If any column keys are specified, other columns without a key provided are assumed
+//! // to be unencrypted
+//! let footer_key = "0123456789012345".as_bytes(); // Keys are 128 bits (16 bytes)
+//! let column_1_key = "1234567890123450".as_bytes();
+//! let column_2_key = "1234567890123451".as_bytes();
+//!
+//! let decryption_properties = FileDecryptionProperties::builder(footer_key.to_vec())
+//! .with_column_key("double_field", column_1_key.to_vec())
+//! .with_column_key("float_field", column_2_key.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let file_metadata = reader_metadata.metadata().file_metadata();
+//! println!("File has {} rows.", file_metadata.num_rows());
Review Comment:
I think a more common pattern in rust crates is to use `assert_eq!` rather than println! -- it then contains the actual expected output in the example and is verified by the docs test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994484025
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
Review Comment:
Ha, it seems that sometimes docs are [tested without encryption](https://github.com/apache/arrow-rs/actions/runs/13847026024/job/38747477104?pr=7283) so this would fail. I'm reintroducing the `cfg` check for now.
However it's not clear to me if the docs that will ultimately be produced will include this example or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#issuecomment-2722447876
@alamb this now covers two reading examples. We can either add it now or wait for the writing path and add it as well.
cc @adamreeve
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994455501
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
Review Comment:
If I understand correctly this does the equivalent of:
```rust
#![cfg(feature = "encryption")]
//! ```rust
#![cfg(not(feature = "encryption"))]
//! ```ignore
```
I noticed that docs generation fails if encryption is not enabled, however I tested now and on CI appears to have encryption enabled when testing docs so I'll remove these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994601568
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,57 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
Review Comment:
Can we add a note here that this requires the experimental `encryption` feature to be enabled?
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,57 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // Define the AES encryption keys required required for decrypting the footer metadata
+//! // and column-specific data. If only a footer key is used then it is assumed that the
+//! // file uses uniform encryption and all columns are encrypted with the footer key.
+//! // If any column keys are specified, other columns without a key provided are assumed
+//! // to be unencrypted
+//! let footer_key = "0123456789012345".as_bytes(); // Keys are 128 bits (16 bytes)
+//! let column_1_key = "1234567890123450".as_bytes();
+//! let column_2_key = "1234567890123451".as_bytes();
+//!
+//! let decryption_properties = FileDecryptionProperties::builder(footer_key.to_vec())
+//! .with_column_key("double_field", column_1_key.to_vec())
+//! .with_column_key("float_field", column_2_key.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
Review Comment:
I think this can be simplified and tidied up a little, we don't need to repeat some of the details that are in examples above:
```suggestion
//! let file_metadata = reader_metadata.metadata().file_metadata();
//! println!("File has {} rows.", file_metadata.num_rows());
//!
//! let mut reader = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap().build().unwrap();
//!
//! let record_batch = reader.next().unwrap().unwrap();
//! println!("Read {} records.", record_batch.num_rows());
//! ```
```
(I also added a closing ```, although the docs seemed to build fine without it?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994490275
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/uniform_encryption.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! let key_code: &[u8] = "0123456789012345".as_bytes();
+//! let decryption_properties = FileDecryptionProperties::builder(key_code.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
+//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
Review Comment:
Removed the uniform example.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994484025
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
Review Comment:
Ha, it seems that sometimes docs are [tested without encryption](https://github.com/apache/arrow-rs/actions/runs/13847026024/job/38747477104?pr=7283) so this would fail. I'm reintroducing the `cfg_attr ` check for now.
However it's not clear to me if the docs that will ultimately be produced will include this example or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994347289
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/uniform_encryption.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! let key_code: &[u8] = "0123456789012345".as_bytes();
+//! let decryption_properties = FileDecryptionProperties::builder(key_code.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
+//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
Review Comment:
I don't know that we need two examples, it might be enough to show the non-uniform encryption example and just have a comment that if no column keys are specified, its assumed that uniform encryption is used and all columns are encrypted with the footer key?
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/uniform_encryption.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! let key_code: &[u8] = "0123456789012345".as_bytes();
+//! let decryption_properties = FileDecryptionProperties::builder(key_code.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
+//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // There is always a footer key even with a plaintext footer,
+//! // but this is used for signing the footer.
Review Comment:
I don't think this comment is that relevant, this isn't a plaintext footer file and plaintext isn't the default. Maybe we could just describe what's happening in a bit more detail and provide some context, eg: "Define the AES encryption keys required required for decrypting the footer metadata and column-specific data. If only a footer key is used then it is assumed that the file uses uniform encryption and all columns are encrypted with the footer key. If any column keys are specified, other columns without a key provided are assumed to be unencrypted"
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
Review Comment:
I'm not that familiar with Rust documentation and doc tests, does cfg_attr mean that the example will only be included in the docs if the encryption feature is enabled, or is that only controlling whether the example is tested? Do we want to always show the example but document that it requires the encryption feature (and that the feature is experimental)?
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/uniform_encryption.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! let key_code: &[u8] = "0123456789012345".as_bytes();
+//! let decryption_properties = FileDecryptionProperties::builder(key_code.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
+//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // There is always a footer key even with a plaintext footer,
+//! // but this is used for signing the footer.
+//! let footer_key = "0123456789012345".as_bytes(); // 128bit/16
Review Comment:
```suggestion
//! let footer_key = "0123456789012345".as_bytes(); // Keys are 128 bits (16 bytes)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994169669
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
Review Comment:
@alamb does this look like a good idea?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1995385611
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,57 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#issuecomment-2725771229
MSRV check is unrelated, see:
- https://github.com/apache/arrow-rs/issues/7289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1994489559
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,96 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/uniform_encryption.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! let key_code: &[u8] = "0123456789012345".as_bytes();
+//! let decryption_properties = FileDecryptionProperties::builder(key_code.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let metadata = reader_metadata.metadata();
+//! let file_metadata = metadata.file_metadata();
+//!
+//! println!("Read {} rows.", file_metadata.num_rows());
+//!
+//! let builder = ParquetRecordBatchReaderBuilder::try_new_with_options(file, options).unwrap();
+//! println!("Converted arrow schema is: {}", builder.schema());
+//!
+//! let mut reader = builder.build().unwrap();
+//!
+//! let record_batch = reader.next().unwrap().unwrap();
+//!
+//! println!("Read {} records.", record_batch.num_rows());
+//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // There is always a footer key even with a plaintext footer,
+//! // but this is used for signing the footer.
Review Comment:
Indeed. Used your comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#issuecomment-2723037385
Thanks for the review @adamreeve ! I'veused your suggestions. I'm not sure what to do about the comments, perhaps @alamb has a better suggestion (https://github.com/apache/arrow-rs/pull/7283#discussion_r1994169669).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283#discussion_r1996252929
##########
parquet/src/arrow/mod.rs:
##########
@@ -93,6 +93,55 @@
//!
//! println!("Read {} records.", record_batch.num_rows());
//! ```
+//!
+//! # Example of reading non-uniformly encrypted parquet file into arrow record batch
+//!
+//! Note: This requires the experimental `encryption` feature to be enabled at compile time.
+//!
+//!
+#![cfg_attr(feature = "encryption", doc = "```rust")]
+#![cfg_attr(not(feature = "encryption"), doc = "```ignore")]
+//! # use arrow_array::{Int32Array, ArrayRef};
+//! # use arrow_array::{types, RecordBatch};
+//! # use parquet::arrow::arrow_reader::{
+//! # ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+//! # };
+//! # use arrow_array::cast::AsArray;
+//! # use parquet::file::metadata::ParquetMetaData;
+//! # use tempfile::tempfile;
+//! # use std::fs::File;
+//! # use parquet::encryption::decrypt::FileDecryptionProperties;
+//! # let test_data = arrow::util::test_util::parquet_test_data();
+//! # let path = format!("{test_data}/encrypt_columns_and_footer.parquet.encrypted");
+//! #
+//! let file = File::open(path).unwrap();
+//!
+//! // Define the AES encryption keys required required for decrypting the footer metadata
+//! // and column-specific data. If only a footer key is used then it is assumed that the
+//! // file uses uniform encryption and all columns are encrypted with the footer key.
+//! // If any column keys are specified, other columns without a key provided are assumed
+//! // to be unencrypted
+//! let footer_key = "0123456789012345".as_bytes(); // Keys are 128 bits (16 bytes)
+//! let column_1_key = "1234567890123450".as_bytes();
+//! let column_2_key = "1234567890123451".as_bytes();
+//!
+//! let decryption_properties = FileDecryptionProperties::builder(footer_key.to_vec())
+//! .with_column_key("double_field", column_1_key.to_vec())
+//! .with_column_key("float_field", column_2_key.to_vec())
+//! .build()
+//! .unwrap();
+//!
+//! let options =
+//! ArrowReaderOptions::default().with_file_decryption_properties(decryption_properties);
+//! let reader_metadata = ArrowReaderMetadata::load(&file, options.clone()).unwrap();
+//! let file_metadata = reader_metadata.metadata().file_metadata();
+//! println!("File has {} rows.", file_metadata.num_rows());
Review Comment:
I took the liberty of doing this in [97ecf40](https://github.com/apache/arrow-rs/pull/7283/commits/97ecf405b1f3b110f8e233db9ccb6cb5f4f33fff)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Add example for how to read encrypted parquet files [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #7283:
URL: https://github.com/apache/arrow-rs/pull/7283
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org