You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2023/07/09 03:06:49 UTC
[GitHub] [arrow] wgtmac opened a new pull request, #36574: GH-34950: [C++][Parquet] Support encryption for page index
wgtmac opened a new pull request, #36574:
URL: https://github.com/apache/arrow/pull/36574
### Rationale for this change
Parquet modular encryption requires page index to be encrypted if the column chunk is encrypted. This feature is missing for now.
### What changes are included in this PR?
Support both encryption and decryption to parquet page index.
### Are these changes tested?
WIP.
### Are there any user-facing changes?
NO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278319593
##########
cpp/src/parquet/page_index.cc:
##########
@@ -744,9 +802,13 @@ class PageIndexBuilderImpl final : public PageIndexBuilder {
for (size_t column = 0; column < num_columns; ++column) {
const auto& column_page_index_builder = row_group_page_index_builders[column];
if (column_page_index_builder != nullptr) {
+ /// Get encryptor if encryption is enabled.
+ auto encryptor = GetColumnMetaEncryptor<Builder>(static_cast<int>(row_group),
Review Comment:
please spell out type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302547704
##########
cpp/src/parquet/page_index.cc:
##########
@@ -830,14 +894,13 @@ RowGroupIndexReadRange PageIndexReader::DeterminePageIndexRangesInRowGroup(
// ----------------------------------------------------------------------
// Public factory functions
-std::unique_ptr<ColumnIndex> ColumnIndex::Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties) {
+std::unique_ptr<ColumnIndex> ColumnIndex::Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties, const std::shared_ptr<Decryptor>& decryptor) {
format::ColumnIndex column_index;
ThriftDeserializer deserializer(properties);
deserializer.DeserializeMessage(reinterpret_cast<const uint8_t*>(serialized_index),
Review Comment:
Fixed. Please take a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278318263
##########
cpp/src/parquet/page_index.h:
##########
@@ -38,10 +41,10 @@ class RowGroupPageIndexReader;
class PARQUET_EXPORT ColumnIndex {
public:
/// \brief Create a ColumnIndex from a serialized thrift message.
- static std::unique_ptr<ColumnIndex> Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties);
+ static std::unique_ptr<ColumnIndex> Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties,
+ const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
Review Comment:
can default parameter be avoided?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1705470553
> @wgtmac Do the encryption data files in https://github.com/apache/parquet-testing/tree/master/data have a page index? If not, can we add some in a subsequent issue/PR? And then add tests against them.
Sure, let me check if these test files can be safely replaced by new files with page index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302678111
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
Review Comment:
This merely says "Encrypt two columns"...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1701279404
> @wgtmac Sorry for the delay, but it seems there is a conflict now, can you resolve it?
No problem at all! Resolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1701018857
@wgtmac Sorry for the delay, but it seems there is a conflict now, can you resolve it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1640544693
@pitrou @emkornfield Please take a look when you have time.
cc @ggershinsky
Also cc @thamht4190 @majetideepak as you have contributed to the encryption in parquet-cpp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278318429
##########
cpp/src/parquet/page_index.h:
##########
@@ -124,9 +127,10 @@ struct PARQUET_EXPORT PageLocation {
class PARQUET_EXPORT OffsetIndex {
public:
/// \brief Create a OffsetIndex from a serialized thrift message.
- static std::unique_ptr<OffsetIndex> Make(const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties);
+ static std::unique_ptr<OffsetIndex> Make(
+ const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties,
+ const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
Review Comment:
same comments as above on signature.
##########
cpp/src/parquet/page_index.h:
##########
@@ -283,7 +287,9 @@ class PARQUET_EXPORT ColumnIndexBuilder {
/// not write any data to the sink.
///
/// \param[out] sink output stream to write the serialized message.
- virtual void WriteTo(::arrow::io::OutputStream* sink) const = 0;
+ /// \param[in] encryptor encryptor to encrypt the serialized column index.
+ virtual void WriteTo(::arrow::io::OutputStream* sink,
Review Comment:
same comments as above on signature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278318005
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +512,177 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ std::string file,
Review Comment:
oh, I see this is still test code, maybe less important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302675230
##########
cpp/src/parquet/page_index.h:
##########
@@ -186,8 +191,7 @@ class PARQUET_EXPORT PageIndexReader {
/// that creates this PageIndexReader.
static std::shared_ptr<PageIndexReader> Make(
::arrow::io::RandomAccessFile* input, std::shared_ptr<FileMetaData> file_metadata,
- const ReaderProperties& properties,
- std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);
+ const ReaderProperties& properties, InternalFileDecryptor* file_decryptor);
Review Comment:
My question is more about `std::shared_ptr<Decryptor>` vs. `InternalFileDecryptor*`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1292913520
##########
cpp/src/parquet/page_index.h:
##########
@@ -38,10 +41,10 @@ class RowGroupPageIndexReader;
class PARQUET_EXPORT ColumnIndex {
public:
/// \brief Create a ColumnIndex from a serialized thrift message.
- static std::unique_ptr<ColumnIndex> Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties);
+ static std::unique_ptr<ColumnIndex> Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties,
+ const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
Review Comment:
Most of encryptor & decryptor input parameters in other places are default to nullptr. But yes, removing default parameters here looks better.
##########
cpp/src/parquet/page_index.h:
##########
@@ -38,10 +41,10 @@ class RowGroupPageIndexReader;
class PARQUET_EXPORT ColumnIndex {
public:
/// \brief Create a ColumnIndex from a serialized thrift message.
- static std::unique_ptr<ColumnIndex> Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties);
+ static std::unique_ptr<ColumnIndex> Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties,
+ const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
Review Comment:
The decryptor will not be a member of returned `ColumnIndex` object. Instead, it is passed to ThriftDeserializer::DeserializeMessage as an input parameter:
```
template <class T>
void DeserializeMessage(const uint8_t* buf, uint32_t* len, T* deserialized_msg,
const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
```
So I think it is better to use `const std::shared_ptr<Decryptor>&`. Let me remove the default value.
##########
cpp/src/parquet/page_index.cc:
##########
@@ -744,9 +802,13 @@ class PageIndexBuilderImpl final : public PageIndexBuilder {
for (size_t column = 0; column < num_columns; ++column) {
const auto& column_page_index_builder = row_group_page_index_builders[column];
if (column_page_index_builder != nullptr) {
+ /// Get encryptor if encryption is enabled.
+ auto encryptor = GetColumnMetaEncryptor<Builder>(static_cast<int>(row_group),
Review Comment:
Do you mean `static_cast<int>`? I think this can avoid two lines in the function call.
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
+ if (file_decryptor_ == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ // The column is encrypted with footer key
+ decryptor = file_decryptor_->GetFooterDecryptorForColumnMeta();
+ } else {
+ // The column is encrypted with its own key
+ const std::string column_key_metadata = crypto_metadata->key_metadata();
Review Comment:
Yes, `column_key_metadata` can but `column_path` below cannot.
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
Review Comment:
Changed.
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
+ if (file_decryptor_ == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ // The column is encrypted with footer key
+ decryptor = file_decryptor_->GetFooterDecryptorForColumnMeta();
+ } else {
+ // The column is encrypted with its own key
+ const std::string column_key_metadata = crypto_metadata->key_metadata();
+ const std::string column_path = crypto_metadata->path_in_schema()->ToDotString();
+ decryptor =
+ file_decryptor_->GetColumnMetaDecryptor(column_path, column_key_metadata);
+ }
+ ARROW_DCHECK(!decryptor->file_aad().empty());
Review Comment:
This should be an internal bug and other places are already using DCHECK for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302534492
##########
cpp/src/parquet/page_index.h:
##########
@@ -186,8 +191,7 @@ class PARQUET_EXPORT PageIndexReader {
/// that creates this PageIndexReader.
static std::shared_ptr<PageIndexReader> Make(
::arrow::io::RandomAccessFile* input, std::shared_ptr<FileMetaData> file_metadata,
- const ReaderProperties& properties,
- std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);
+ const ReaderProperties& properties, InternalFileDecryptor* file_decryptor);
Review Comment:
For `RandomAccessFile*` and `InternalFileDecryptor*`, I want to explicitly state that returned PageIndexReader does not own these objects and they are owned by the parquet file reader. However, it is unfortunate that no one owns `file_metadata` so it must be a `std::shared_ptr<FileMetaData>`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1705395907
@wgtmac Do the encryption data files in https://github.com/apache/parquet-testing/tree/master/data have a page index? If not, can we add some in a subsequent issue/PR? And then add tests against them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278320504
##########
cpp/src/parquet/page_index.cc:
##########
@@ -744,9 +802,13 @@ class PageIndexBuilderImpl final : public PageIndexBuilder {
for (size_t column = 0; column < num_columns; ++column) {
const auto& column_page_index_builder = row_group_page_index_builders[column];
if (column_page_index_builder != nullptr) {
+ /// Get encryptor if encryption is enabled.
+ auto encryptor = GetColumnMetaEncryptor<Builder>(static_cast<int>(row_group),
Review Comment:
Is there a reason to cast here, versus casting within the function call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1727025332
I have re-generate encrypted files (with page index enabled) in the parquet-testing repo: https://github.com/apache/parquet-testing/pull/39
All comments are addressed. Please take a look again. Thanks! @pitrou @emkornfield
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1335920339
##########
cpp/src/parquet/page_index.h:
##########
@@ -25,14 +25,13 @@
namespace parquet {
-class ColumnDescriptor;
-class EncodedStatistics;
-class FileMetaData;
+class Decryptor;
+class Encryptor;
class InternalFileDecryptor;
+class InternalFileEncryptor;
Review Comment:
It would probably be useful to have a `encryption/type_fwd.h` to centralize all such forward declarations?
##########
cpp/src/parquet/encryption/internal_file_decryptor.cc:
##########
@@ -215,4 +217,53 @@ std::shared_ptr<Decryptor> InternalFileDecryptor::GetColumnDecryptor(
return column_data_map_[column_path];
}
+namespace {
+
+std::shared_ptr<Decryptor> GetColumnDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, InternalFileDecryptor* file_decryptor,
+ const std::function<std::shared_ptr<Decryptor>(
+ InternalFileDecryptor* file_decryptor, const std::string& column_path,
+ const std::string& column_key_metadata, const std::string& aad)>& func) {
+ if (crypto_metadata == nullptr) {
+ return nullptr;
+ }
+
+ if (file_decryptor == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ return file_decryptor->GetFooterDecryptorForColumnMeta();
Review Comment:
Hmm... it should be `GetFooterDecryptorForColumnData` if called from `GetColumnDataDecryptor`, shouldn't it?
##########
cpp/src/parquet/encryption/internal_file_decryptor.cc:
##########
@@ -215,4 +217,53 @@ std::shared_ptr<Decryptor> InternalFileDecryptor::GetColumnDecryptor(
return column_data_map_[column_path];
}
+namespace {
+
+std::shared_ptr<Decryptor> GetColumnDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, InternalFileDecryptor* file_decryptor,
+ const std::function<std::shared_ptr<Decryptor>(
+ InternalFileDecryptor* file_decryptor, const std::string& column_path,
+ const std::string& column_key_metadata, const std::string& aad)>& func) {
+ if (crypto_metadata == nullptr) {
+ return nullptr;
+ }
+
+ if (file_decryptor == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ return file_decryptor->GetFooterDecryptorForColumnMeta();
Review Comment:
I'm curious why this doesn't fail the tests, though.
##########
cpp/src/parquet/encryption/internal_file_decryptor.cc:
##########
@@ -215,4 +217,53 @@ std::shared_ptr<Decryptor> InternalFileDecryptor::GetColumnDecryptor(
return column_data_map_[column_path];
}
+namespace {
+
+std::shared_ptr<Decryptor> GetColumnDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, InternalFileDecryptor* file_decryptor,
+ const std::function<std::shared_ptr<Decryptor>(
+ InternalFileDecryptor* file_decryptor, const std::string& column_path,
+ const std::string& column_key_metadata, const std::string& aad)>& func) {
+ if (crypto_metadata == nullptr) {
+ return nullptr;
+ }
+
+ if (file_decryptor == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ return file_decryptor->GetFooterDecryptorForColumnMeta();
Review Comment:
Ah, this might be because the only distinction between data and metadata is when selecting the actual cipher, and only if the selected algorithm id is not already `AES_GCM_V1`. Here is for example `AesDecryptor`:
```c++
AesDecryptor::AesDecryptorImpl::AesDecryptorImpl(ParquetCipher::type alg_id, int key_len,
bool metadata, bool contains_length) {
ctx_ = nullptr;
length_buffer_length_ = contains_length ? kBufferSizeLength : 0;
ciphertext_size_delta_ = length_buffer_length_ + kNonceLength;
if (metadata || (ParquetCipher::AES_GCM_V1 == alg_id)) {
aes_mode_ = kGcmMode;
ciphertext_size_delta_ += kGcmTagLength;
} else {
aes_mode_ = kCtrMode;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1739515438
> @wgtmac It looks like your latest push broke compilation, see CI failures.
I just found that and fixed. Sorry for the stupid mistake.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1338814429
##########
cpp/src/parquet/encryption/internal_file_decryptor.cc:
##########
@@ -215,4 +217,53 @@ std::shared_ptr<Decryptor> InternalFileDecryptor::GetColumnDecryptor(
return column_data_map_[column_path];
}
+namespace {
+
+std::shared_ptr<Decryptor> GetColumnDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, InternalFileDecryptor* file_decryptor,
+ const std::function<std::shared_ptr<Decryptor>(
+ InternalFileDecryptor* file_decryptor, const std::string& column_path,
+ const std::string& column_key_metadata, const std::string& aad)>& func) {
+ if (crypto_metadata == nullptr) {
+ return nullptr;
+ }
+
+ if (file_decryptor == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ return file_decryptor->GetFooterDecryptorForColumnMeta();
Review Comment:
Good catch! Sorry for the mistake made by the copy-and-paste.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278318416
##########
cpp/src/parquet/page_index.h:
##########
@@ -38,10 +41,10 @@ class RowGroupPageIndexReader;
class PARQUET_EXPORT ColumnIndex {
public:
/// \brief Create a ColumnIndex from a serialized thrift message.
- static std::unique_ptr<ColumnIndex> Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties);
+ static std::unique_ptr<ColumnIndex> Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties,
+ const std::shared_ptr<Decryptor>& decryptor = NULLPTR);
Review Comment:
I think if we are sharing ownership the shared_ptr should be passed by value. If we do not intend to share ownership pass by raw pointer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1299833893
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
Review Comment:
I don't understand: only columns 4 and 5 are encrypted? Is this documented somewhere?
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
+ } else {
+ need_columns = {0, 1, 2, 3, 4, 5, 6, 7};
+ }
+
+ // Provide hint of requested columns to avoid accessing encrypted columns without
+ // decryption properties.
+ page_index_reader->WillNeed(
+ need_row_groups, need_columns,
+ PageIndexSelection{/*column_index=*/true, /*offset_index=*/true});
+
+ // Iterate over all the RowGroups in the file.
+ for (int r = 0; r < num_row_groups; ++r) {
+ auto row_group_page_index_reader = page_index_reader->RowGroup(r);
+ ASSERT_NE(row_group_page_index_reader, nullptr);
+
+ for (int c = 0; c < num_columns; ++c) {
+ // Skip reading encrypted columns without decryption properties.
+ if (file_decryption_properties == nullptr && (c == 4 || c == 5)) {
+ continue;
+ }
+
+ constexpr size_t kExpectedNumPages = 1;
+
+ // Check offset index.
+ auto offset_index = row_group_page_index_reader->GetOffsetIndex(c);
+ ASSERT_NE(offset_index, nullptr);
+ ASSERT_EQ(offset_index->page_locations().size(), kExpectedNumPages);
+ const auto& first_page = offset_index->page_locations()[0];
+ ASSERT_EQ(first_page.first_row_index, 0);
+ ASSERT_GT(first_page.compressed_page_size, 0);
+
+ // Int96 column does not have column index.
+ if (c == 3) {
+ continue;
+ }
+
+ // Check column index
+ auto column_index = row_group_page_index_reader->GetColumnIndex(c);
+ ASSERT_NE(column_index, nullptr);
+ ASSERT_EQ(column_index->null_pages().size(), kExpectedNumPages);
+ ASSERT_EQ(column_index->null_pages()[0], false);
+ ASSERT_EQ(column_index->encoded_min_values().size(), kExpectedNumPages);
+ ASSERT_EQ(column_index->encoded_max_values().size(), kExpectedNumPages);
+ ASSERT_TRUE(column_index->has_null_counts());
+
+ switch (c) {
+ case 0: {
+ auto bool_column_index =
+ std::dynamic_pointer_cast<BoolColumnIndex>(column_index);
+ ASSERT_NE(bool_column_index, nullptr);
+ constexpr int kExpectedNullCount = 0;
+ constexpr bool kExpectedMin = false;
+ constexpr bool kExpectedMax = true;
+ ASSERT_EQ(bool_column_index->null_counts().at(0), kExpectedNullCount);
+ ASSERT_EQ(static_cast<bool>(bool_column_index->min_values()[0]), kExpectedMin);
+ ASSERT_EQ(static_cast<bool>(bool_column_index->max_values()[0]), kExpectedMax);
Review Comment:
I'm sure you can write a (templated?) helper function to avoid repeating yourself here.
For example:
```c++
template <typename DType, typename c_type = typename DType::c_type>
void AssertColumnIndex(
const ColumnIndex&,
const std::vector<int64_t>& expected_null_counts,
const std::vector<c_type>& expected_min_values,
const std::vector<c_type>& expected_max_values);
```
##########
cpp/src/parquet/page_index.h:
##########
@@ -186,8 +191,7 @@ class PARQUET_EXPORT PageIndexReader {
/// that creates this PageIndexReader.
static std::shared_ptr<PageIndexReader> Make(
::arrow::io::RandomAccessFile* input, std::shared_ptr<FileMetaData> file_metadata,
- const ReaderProperties& properties,
- std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);
+ const ReaderProperties& properties, InternalFileDecryptor* file_decryptor);
Review Comment:
Why do some APIs take a raw pointer and others a shared_ptr? Is there a reason for the inconsistency?
##########
cpp/src/parquet/page_index.cc:
##########
@@ -830,14 +894,13 @@ RowGroupIndexReadRange PageIndexReader::DeterminePageIndexRangesInRowGroup(
// ----------------------------------------------------------------------
// Public factory functions
-std::unique_ptr<ColumnIndex> ColumnIndex::Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties) {
+std::unique_ptr<ColumnIndex> ColumnIndex::Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties, const std::shared_ptr<Decryptor>& decryptor) {
format::ColumnIndex column_index;
ThriftDeserializer deserializer(properties);
deserializer.DeserializeMessage(reinterpret_cast<const uint8_t*>(serialized_index),
Review Comment:
Same for `SerializeMessage`, incidentally.
##########
cpp/src/parquet/page_index.cc:
##########
@@ -725,6 +761,29 @@ class PageIndexBuilderImpl final : public PageIndexBuilder {
}
}
+ template <typename Builder>
Review Comment:
Instead of templating this on the builder type, why wouldn't you simply pass the `module_type` as a function argument?
##########
cpp/src/parquet/page_index.cc:
##########
@@ -830,14 +894,13 @@ RowGroupIndexReadRange PageIndexReader::DeterminePageIndexRangesInRowGroup(
// ----------------------------------------------------------------------
// Public factory functions
-std::unique_ptr<ColumnIndex> ColumnIndex::Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties) {
+std::unique_ptr<ColumnIndex> ColumnIndex::Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties, const std::shared_ptr<Decryptor>& decryptor) {
format::ColumnIndex column_index;
ThriftDeserializer deserializer(properties);
deserializer.DeserializeMessage(reinterpret_cast<const uint8_t*>(serialized_index),
Review Comment:
Can `DeserializeMessage` be updated to take a raw `Decryptor*`? It doesn't need to take ownership...
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
Review Comment:
Aren't you reimplementing `std::iota` here?
##########
cpp/src/parquet/page_index.h:
##########
@@ -332,7 +340,8 @@ class PARQUET_EXPORT OffsetIndexBuilder {
class PARQUET_EXPORT PageIndexBuilder {
public:
/// \brief API convenience to create a PageIndexBuilder.
- static std::unique_ptr<PageIndexBuilder> Make(const SchemaDescriptor* schema);
+ static std::unique_ptr<PageIndexBuilder> Make(const SchemaDescriptor* schema,
+ InternalFileEncryptor* file_encryptor);
Review Comment:
These APIs are probably not widely-used, but they are still public and we should minimize API compatibility breakage. Can we make the encryption/decryption arguments optional? (or have overloads without them)
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
+ } else {
+ need_columns = {0, 1, 2, 3, 4, 5, 6, 7};
+ }
+
+ // Provide hint of requested columns to avoid accessing encrypted columns without
+ // decryption properties.
+ page_index_reader->WillNeed(
+ need_row_groups, need_columns,
+ PageIndexSelection{/*column_index=*/true, /*offset_index=*/true});
+
+ // Iterate over all the RowGroups in the file.
+ for (int r = 0; r < num_row_groups; ++r) {
+ auto row_group_page_index_reader = page_index_reader->RowGroup(r);
+ ASSERT_NE(row_group_page_index_reader, nullptr);
+
+ for (int c = 0; c < num_columns; ++c) {
+ // Skip reading encrypted columns without decryption properties.
Review Comment:
Instead of skipping, shouldn't we check that reading said columns fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1315021039
##########
cpp/src/parquet/page_index.h:
##########
@@ -26,9 +26,12 @@
namespace parquet {
class ColumnDescriptor;
Review Comment:
Can we perhaps use the relevant `type_fwd.h` includes instead of repeating all those fw-declarations?
##########
cpp/src/parquet/page_index.h:
##########
@@ -43,6 +46,12 @@ class PARQUET_EXPORT ColumnIndex {
uint32_t index_len,
const ReaderProperties& properties);
+ static std::unique_ptr<ColumnIndex> Make(const ColumnDescriptor& descr,
+ const void* serialized_index,
+ uint32_t index_len,
+ const ReaderProperties& properties,
+ Decryptor* decryptor);
Review Comment:
Perhaps we can simply `decryptor` an optional argument (defaulting to `NULLPTR`) to avoid adding overloads :-)
##########
cpp/src/parquet/page_index.cc:
##########
@@ -308,6 +312,34 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ if (crypto_metadata == nullptr) {
+ return nullptr;
+ }
+ if (file_decryptor_ == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ // The column is encrypted with footer key
+ decryptor = file_decryptor_->GetFooterDecryptorForColumnMeta();
+ } else {
+ // The column is encrypted with its own key
+ const std::string& column_key_metadata = crypto_metadata->key_metadata();
+ const std::string column_path = crypto_metadata->path_in_schema()->ToDotString();
+ decryptor =
+ file_decryptor_->GetColumnMetaDecryptor(column_path, column_key_metadata);
+ }
+ ARROW_DCHECK(!decryptor->file_aad().empty());
+ decryptor->UpdateAad(encryption::CreateModuleAad(decryptor->file_aad(), module_type,
+ row_group_ordinal_, column_ordinal,
+ kNonPageOrdinal));
+ return decryptor;
+ }
Review Comment:
This is not far from the code in `GetColumnPageReader` (in `file_reader.cc`). It would be nice to factor out those per-column decryption facilities, either in this PR or in a subsequent issue, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1739510364
@wgtmac It looks like your latest push broke compilation, see CI failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278317935
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +512,177 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ std::string file,
+ std::shared_ptr<FileDecryptionProperties> file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties.get());
+
+ file_reader->Close();
Review Comment:
can file_reader->Close throw an error, is there a leak if source is not closed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278325164
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
+ if (file_decryptor_ == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ // The column is encrypted with footer key
+ decryptor = file_decryptor_->GetFooterDecryptorForColumnMeta();
+ } else {
+ // The column is encrypted with its own key
+ const std::string column_key_metadata = crypto_metadata->key_metadata();
+ const std::string column_path = crypto_metadata->path_in_schema()->ToDotString();
+ decryptor =
+ file_decryptor_->GetColumnMetaDecryptor(column_path, column_key_metadata);
+ }
+ ARROW_DCHECK(!decryptor->file_aad().empty());
Review Comment:
Does this represent an internal bug, or some sort of configuration error? If the latter an exception should probably be raised.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278327616
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
Review Comment:
consider changing this to a guard to remove one level of nesting in the main function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302503911
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
Review Comment:
Good catch! Let me change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302546880
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
+ } else {
+ need_columns = {0, 1, 2, 3, 4, 5, 6, 7};
+ }
+
+ // Provide hint of requested columns to avoid accessing encrypted columns without
+ // decryption properties.
+ page_index_reader->WillNeed(
+ need_row_groups, need_columns,
+ PageIndexSelection{/*column_index=*/true, /*offset_index=*/true});
+
+ // Iterate over all the RowGroups in the file.
+ for (int r = 0; r < num_row_groups; ++r) {
+ auto row_group_page_index_reader = page_index_reader->RowGroup(r);
+ ASSERT_NE(row_group_page_index_reader, nullptr);
+
+ for (int c = 0; c < num_columns; ++c) {
+ // Skip reading encrypted columns without decryption properties.
+ if (file_decryption_properties == nullptr && (c == 4 || c == 5)) {
+ continue;
+ }
+
+ constexpr size_t kExpectedNumPages = 1;
+
+ // Check offset index.
+ auto offset_index = row_group_page_index_reader->GetOffsetIndex(c);
+ ASSERT_NE(offset_index, nullptr);
+ ASSERT_EQ(offset_index->page_locations().size(), kExpectedNumPages);
+ const auto& first_page = offset_index->page_locations()[0];
+ ASSERT_EQ(first_page.first_row_index, 0);
+ ASSERT_GT(first_page.compressed_page_size, 0);
+
+ // Int96 column does not have column index.
+ if (c == 3) {
+ continue;
+ }
+
+ // Check column index
+ auto column_index = row_group_page_index_reader->GetColumnIndex(c);
+ ASSERT_NE(column_index, nullptr);
+ ASSERT_EQ(column_index->null_pages().size(), kExpectedNumPages);
+ ASSERT_EQ(column_index->null_pages()[0], false);
+ ASSERT_EQ(column_index->encoded_min_values().size(), kExpectedNumPages);
+ ASSERT_EQ(column_index->encoded_max_values().size(), kExpectedNumPages);
+ ASSERT_TRUE(column_index->has_null_counts());
+
+ switch (c) {
+ case 0: {
+ auto bool_column_index =
+ std::dynamic_pointer_cast<BoolColumnIndex>(column_index);
+ ASSERT_NE(bool_column_index, nullptr);
+ constexpr int kExpectedNullCount = 0;
+ constexpr bool kExpectedMin = false;
+ constexpr bool kExpectedMax = true;
+ ASSERT_EQ(bool_column_index->null_counts().at(0), kExpectedNullCount);
+ ASSERT_EQ(static_cast<bool>(bool_column_index->min_values()[0]), kExpectedMin);
+ ASSERT_EQ(static_cast<bool>(bool_column_index->max_values()[0]), kExpectedMax);
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1739426020
> The CI failures are having a hard time with LLVM.
Yes, it's been happening for some time unfortunately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1340298780
##########
cpp/src/parquet/page_index.h:
##########
@@ -283,7 +280,8 @@ class PARQUET_EXPORT ColumnIndexBuilder {
/// not write any data to the sink.
///
/// \param[out] sink output stream to write the serialized message.
- virtual void WriteTo(::arrow::io::OutputStream* sink) const = 0;
+ /// \param[in] encryptor encryptor to encrypt the serialized column index.
+ virtual void WriteTo(::arrow::io::OutputStream* sink, Encryptor* encryptor) const = 0;
Review Comment:
It was suggested by @emkornfield in the review comments to avoid default parameter as much as possible. However, almost all encryptors/decryptors in the parquet-cpp are default to null. I think here I can make it default to null to be consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1338814935
##########
cpp/src/parquet/page_index.h:
##########
@@ -25,14 +25,13 @@
namespace parquet {
-class ColumnDescriptor;
-class EncodedStatistics;
-class FileMetaData;
+class Decryptor;
+class Encryptor;
class InternalFileDecryptor;
+class InternalFileEncryptor;
Review Comment:
Done. Please take a look again. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1738455561
The CI failures are having a hard time with LLVM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278317788
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +512,177 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ std::string file,
Review Comment:
nit: const-ref
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1656782209
Overall question, it looks like offset index might also get encrypted, is there a reason why? This doesn't seem like it has any identifiable information?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302509826
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
+ } else {
+ need_columns = {0, 1, 2, 3, 4, 5, 6, 7};
+ }
+
+ // Provide hint of requested columns to avoid accessing encrypted columns without
+ // decryption properties.
+ page_index_reader->WillNeed(
+ need_row_groups, need_columns,
+ PageIndexSelection{/*column_index=*/true, /*offset_index=*/true});
+
+ // Iterate over all the RowGroups in the file.
+ for (int r = 0; r < num_row_groups; ++r) {
+ auto row_group_page_index_reader = page_index_reader->RowGroup(r);
+ ASSERT_NE(row_group_page_index_reader, nullptr);
+
+ for (int c = 0; c < num_columns; ++c) {
+ // Skip reading encrypted columns without decryption properties.
Review Comment:
The reason is that PageIndexReader will try to coalesce I/O ranges of requested columns. This requires decryption of column metadata. Without file decryption properties, we cannot even read any column here. So I decided to skip these encrypted columns.
Actually those failing cases are checked in the test here and around: https://github.com/apache/arrow/pull/36574/commits/6741c7213d99ad892b696629c620e96a2d8398bf#diff-371a1f4f383d941e926614b706b04810a795c4e14bd25ef898371dd38dd54582R213
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1340276213
##########
cpp/src/parquet/page_index.h:
##########
@@ -283,7 +280,8 @@ class PARQUET_EXPORT ColumnIndexBuilder {
/// not write any data to the sink.
///
/// \param[out] sink output stream to write the serialized message.
- virtual void WriteTo(::arrow::io::OutputStream* sink) const = 0;
+ /// \param[in] encryptor encryptor to encrypt the serialized column index.
+ virtual void WriteTo(::arrow::io::OutputStream* sink, Encryptor* encryptor) const = 0;
Review Comment:
I'm curious why we don't make `encryptor` optional as in `PageIndexBuilder` below?
##########
cpp/src/parquet/page_index.h:
##########
@@ -322,7 +320,8 @@ class PARQUET_EXPORT OffsetIndexBuilder {
/// \brief Serialize the offset index thrift message.
///
/// \param[out] sink output stream to write the serialized message.
- virtual void WriteTo(::arrow::io::OutputStream* sink) const = 0;
+ /// \param[in] encryptor encryptor to encrypt the serialized offset index.
+ virtual void WriteTo(::arrow::io::OutputStream* sink, Encryptor* encryptor) const = 0;
Review Comment:
Same question here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou merged pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #36574:
URL: https://github.com/apache/arrow/pull/36574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302547704
##########
cpp/src/parquet/page_index.cc:
##########
@@ -830,14 +894,13 @@ RowGroupIndexReadRange PageIndexReader::DeterminePageIndexRangesInRowGroup(
// ----------------------------------------------------------------------
// Public factory functions
-std::unique_ptr<ColumnIndex> ColumnIndex::Make(const ColumnDescriptor& descr,
- const void* serialized_index,
- uint32_t index_len,
- const ReaderProperties& properties) {
+std::unique_ptr<ColumnIndex> ColumnIndex::Make(
+ const ColumnDescriptor& descr, const void* serialized_index, uint32_t index_len,
+ const ReaderProperties& properties, const std::shared_ptr<Decryptor>& decryptor) {
format::ColumnIndex column_index;
ThriftDeserializer deserializer(properties);
deserializer.DeserializeMessage(reinterpret_cast<const uint8_t*>(serialized_index),
Review Comment:
It seems to involve a lot of change. I will try to take a deep look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302681571
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
Review Comment:
Right, checking the code below we can know that `path_to_double_field_` and `path_to_float_field_` are encrypted...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302682856
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
Review Comment:
Fair enough :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302682995
##########
cpp/src/parquet/page_index.h:
##########
@@ -186,8 +191,7 @@ class PARQUET_EXPORT PageIndexReader {
/// that creates this PageIndexReader.
static std::shared_ptr<PageIndexReader> Make(
::arrow::io::RandomAccessFile* input, std::shared_ptr<FileMetaData> file_metadata,
- const ReaderProperties& properties,
- std::shared_ptr<InternalFileDecryptor> file_decryptor = NULLPTR);
+ const ReaderProperties& properties, InternalFileDecryptor* file_decryptor);
Review Comment:
OK. `std::shared_ptr<Decryptor>` is due to the expectation of `DeserializeMessage` call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302499077
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +513,178 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ const std::string& file,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::string exception_msg;
+ parquet::ReaderProperties reader_properties = parquet::default_reader_properties();
+ if (file_decryption_properties) {
+ reader_properties.file_decryption_properties(file_decryption_properties->DeepClone());
+ }
+
+ std::shared_ptr<::arrow::io::RandomAccessFile> source;
+ PARQUET_ASSIGN_OR_THROW(
+ source, ::arrow::io::ReadableFile::Open(file, reader_properties.memory_pool()));
+
+ auto file_reader = parquet::ParquetFileReader::Open(source, reader_properties);
+ CheckPageIndex(file_reader.get(), file_decryption_properties);
+
+ ASSERT_NO_FATAL_FAILURE(file_reader->Close());
+ PARQUET_THROW_NOT_OK(source->Close());
+}
+
+void FileDecryptor::CheckPageIndex(
+ parquet::ParquetFileReader* file_reader,
+ const std::shared_ptr<FileDecryptionProperties>& file_decryption_properties) {
+ std::shared_ptr<PageIndexReader> page_index_reader = file_reader->GetPageIndexReader();
+ ASSERT_NE(page_index_reader, nullptr);
+
+ const std::shared_ptr<parquet::FileMetaData> file_metadata = file_reader->metadata();
+ const int num_row_groups = file_metadata->num_row_groups();
+ const int num_columns = file_metadata->num_columns();
+ ASSERT_EQ(num_columns, 8);
+
+ // We cannot read page index of encrypted columns in the plaintext mode
+ std::vector<int32_t> need_row_groups(num_row_groups);
+ std::for_each(need_row_groups.begin(), need_row_groups.end(),
+ [i = 0](int32_t& v) mutable { v = i++; });
+ std::vector<int32_t> need_columns;
+ if (file_decryption_properties == nullptr) {
+ need_columns = {0, 1, 2, 3, 6, 7};
Review Comment:
Yes, please take a look at this: https://github.com/apache/arrow/blob/0781f1341446b8dbb281ac2910dc5e05ec982851/cpp/src/parquet/encryption/write_configurations_test.cc#L32-L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wgtmac commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1302537048
##########
cpp/src/parquet/page_index.h:
##########
@@ -332,7 +340,8 @@ class PARQUET_EXPORT OffsetIndexBuilder {
class PARQUET_EXPORT PageIndexBuilder {
public:
/// \brief API convenience to create a PageIndexBuilder.
- static std::unique_ptr<PageIndexBuilder> Make(const SchemaDescriptor* schema);
+ static std::unique_ptr<PageIndexBuilder> Make(const SchemaDescriptor* schema,
+ InternalFileEncryptor* file_encryptor);
Review Comment:
OK, I prefer the latter which does not have a default parameter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1739517702
Thanks a lot for this @wgtmac . Hopefully CI will pass now (except for unrelated failures...).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] conbench-apache-arrow[bot] commented on pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #36574:
URL: https://github.com/apache/arrow/pull/36574#issuecomment-1740188599
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 853d8491addff3a10fc40950823a2942bb9fbf98.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/17239111130) has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278321822
##########
cpp/src/parquet/page_index.cc:
##########
@@ -309,6 +313,33 @@ class RowGroupPageIndexReaderImpl : public RowGroupPageIndexReader {
}
}
+ /// Get decryptor if the file is encrypted
+ std::shared_ptr<Decryptor> GetColumnMetaDecryptor(
+ const ColumnCryptoMetaData* crypto_metadata, int column_ordinal,
+ int8_t module_type) const {
+ std::shared_ptr<Decryptor> decryptor;
+ if (crypto_metadata != nullptr) {
+ if (file_decryptor_ == nullptr) {
+ throw ParquetException("RowGroup is noted as encrypted but no file decryptor");
+ }
+ if (crypto_metadata->encrypted_with_footer_key()) {
+ // The column is encrypted with footer key
+ decryptor = file_decryptor_->GetFooterDecryptorForColumnMeta();
+ } else {
+ // The column is encrypted with its own key
+ const std::string column_key_metadata = crypto_metadata->key_metadata();
Review Comment:
can string copies be avoided here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #36574: GH-34950: [C++][Parquet] Support encryption for page index
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #36574:
URL: https://github.com/apache/arrow/pull/36574#discussion_r1278317889
##########
cpp/src/parquet/encryption/test_encryption_util.cc:
##########
@@ -509,4 +512,177 @@ void FileDecryptor::CheckFile(parquet::ParquetFileReader* file_reader,
}
}
+void FileDecryptor::DecryptPageIndex(
+ std::string file,
+ std::shared_ptr<FileDecryptionProperties> file_decryption_properties) {
Review Comment:
nit: pass either as const-ref or as a raw pointer (ownership sharing doesn't seem necessary here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org